Proper sequencing for pp with layers and masks in Gimp

[JBW]

Hi, I have started using layers in Gimp and can do reasonable pp with it. But now I have begun to explore masks and I am getting seriously confused.

What I need is a suggestion or two indicate where a mask should be inserted? Before or after this or that?

As I mentioned I am seriously confused about masks and how to use them.
Brian

[Shadowman]

I guess it depends on how you plan to use it and what for. This link applies to using layered masks for softening skin.

http://www.photoshopessentials.com/p...g/smooth-skin/

It's similar to the one I just followed using Scott Kelby's technique from his book "the Photoshop Elements" series. Quick sequence was:
1. Duplicate layer.
2. Apply an overall blur.
3. Create a new layer.
4. Group blurred and original layer.
5. Apply skin smoothing to new layer.

and so on. The above steps should be similar to the one's in the link. I think it's best to apply a technique that closely resembles what you are trying to achieve. As most of your work involves macro shots, I would look for a tutorial dedicated to that type of photography. Hope this helps.

[ajohnw]

That type of layer work is used to isolate parts of an image to work on them separately rather than the whole image.

There are plenty of tutorials about on various ways of generating layer masks that make it look simple but often they need touching up by hand in practice. Here's one for instance that generates them from various luminosity variations in a shot. The page also has a link to more information on layer masks.

http://www.gimp.org/tutorials/Luminosity_Masks/

Channels crop up due to GIMP's channel to mask feature. Channels are normally just the r g and b values show separately. These too can sometimes be used to form masks of one type or another as say the green channel might have most of the info needed to form the basis of a mask with little additional work. To see these just make the layers invisible and then any of the channels visible. Shift click helps with this sometimes. Try it to see what it does when there are several layers about.

Adjustment layers are something else especially in Adobe products. The GIMP has some too. I've already mentioned a couple but they generally aren't referred to as adjustment layers - just layers that do this and that.

John
-

[Geoff F]

If you are initially confused about Adjustment Layers, Brian, try creating a duplicate layer from the original background image and editing the new layer as required. Don't worry about masks for now, just get the area which you want to change correct.

Then add a mask to the layer. To hide or reveal that layer. Then start to edit the mask so as to selectively apply the effect just where it is needed.

That is a slightly longer way of doing things, and creates a larger file size; but it might be easier to start with.

The main difficulty with masking is to decide which type of mask to use, then remembering to use the correct form of brush. Try to see it as positive or negative editing.

Possibly post some examples of images where you are getting confused.

[JBW]

If you are initially confused about Adjustment Layers, Brian, try creating a duplicate layer from the original background image and editing the new layer as required. Don't worry about masks for now, just get the area which you want to change correct.

Then add a mask to the layer. To hide or reveal that layer. Then start to edit the mask so as to selectively apply the effect just where it is needed.

That is a slightly longer way of doing things, and creates a larger file size; but it might be easier to start with.

The main difficulty with masking is to decide which type of mask to use, then remembering to use the correct form of brush. Try to see it as positive or negative editing.

Possibly post some examples of images where you are getting confused.

Hi Geoff, maybe we could move this up a notch and turn this into some sort of tutorial on layers and masks. I took this shot this morning. It is a jpeg right out of my FufiFilm S4200. Not even cropped. It seems to me to be a pretty good shot that has a lot of challenges for pp? I will work on it in layers and maybe others will have a go at with layers and masks?


This is my best effort using layers in Gimp. After cropping and making a copy of the background I added (in separate layers) a convolution matrix, colour balance,antialias, brightness and contrast, nl filter, unsharpen mask and finally the round corners. Then I hid the background layer and merged the rest. if there is a way to use a mask that would darken the left and right sides it would be nice.

[Stagecoach]

Hi Brian,

Not sure about Gimp but in Elements I would use a 'gradient' mask of the 'radial' type that would allow darkening of a circular area graduating smoothly outwards from the subject. You could of course also use 'linear' gradients in any plane.

I tried it on your original that I did nothing else on other than minor sharpening and toning down the highlights. The radial gradient method is similar to what you can achieve with a vignette but I find it so easy to place the centre exactly where you want it within the image.

Grahame

[ajohnw]

It would take way too much typing to meet your request Brian. The tutorials link I provided gives a good run down on layer masks. The basic principle is that a white layer mask lets nothing through, black lets all through and shades of grey behave accordingly. If due to the set up it works out the other way round - tough, just do it again the other way round. There is a link to a simple explanation off the page on luminosity masks posted and yet another off that page. The reason I linked to the luminosity one is that it's often useful. As the man says - try experimentation. While the main example shows toning at the end he shows what happens if a curve is applied to one of the luminosity levels. Both aspects can be useful to you. He also shows how to generate these mask manually but at the end mentions a plugin that will do this automatically.

Gradients have been mentioned. The GIMP tool is explained here, it can have various shapes.

http://docs.gimp.org/2.2/en/gimp-tool-blend.html

You could simply follow the tutorial and use this rather than the gradient steps he shows. He uses colour in one example you would use black to white. You could also use it on a layer.

There is also a tutorial on facial retouching. Not of direct use but it provides layers based round detail level. Periodically I play around with that to see what it can do on other types of shots. It's possible to heavily sharpen fine detail but it can leave halos which currently I blur out manually. Other adjustments may do something else or the same thing. It can be used to blur detail which is why it's mainly used for facial retouching. The selective sharpening technique I mentioned can also be used as in macro work the main thing that needs this is the subject. It too can be varied, some to the lot, some to the specimen etc.

One of the problems with the GIMP is that I'm sure no one on the planet knows all of the things that can be done with it. The best option is to collect techniques and then figure out how they can be used on different types of shots. When it comes to making full use of masks just like the selective sharpening hand retouching is often needed to make full use of them. Tutorials generally show a shot where it just all works without that.

John
-

[JBW]

John I have been over those tutorials time and again and I just get lost. My mind is not so good in some areas.

[Loose Canon]

Hi Brian,

I understand the confusion. Layer masks are confusing at first. But like anything once you get the hang of it it’s a piece of cake. So no worries! I’ll try to take a shot here and see if I can help make things clear as mud!

Please note that I don’t use Gimp, but I’ve had a cursory look around and masks seem very similar in Photoshop. I think the issue may get even more confused getting advice about using gradient masks, shaped masks, etc. when one is first trying to get a handle on all this. I apologize in advance if I am about to oversimplify but if you are already to this point, maybe someone else isn’t.

Let’s break it down!

So you have your shot where you want it except you want to darken the right and left side. You have merged all layers except the base layer. So you should have two layers in your layer stack.

First, if you wanted to keep all those layers rather than merge to only one there is a nifty command that makes that happen. It is called “stamp visible”. Highlight (click) the top layer in the stack and then find how to “stamp visible” in Gimp (I googled it and know it can be done). What this does is merge all layers into a new layer on top of the stack. A new base layer on top. Since it is a new merged layer the other layers below it will not have any effect on the picture now. You are working with a new merged layer. The advantage is that you can always get back to your other layers if need be. The disadvantage is it increases the file size. But when you are completely done you can always go back and delete any layers you want to lessen the file size.

But since you didn’t do that we’ll go from where you are with the hidden BG layer and the merged layer on top of it.



Remember I’m in Photoshop but you should have something similar showing for your layer stack.

Duplicate the visible merged layer and rename it DarkBG or similar…



Now it is my understanding that Gimp has no “adjustment layers” and I could very well be wrong about that. So you need to use the levels tool on the duplicate layer to make the layer darker. It should make the entire layer darker but no worries. Just adjust it darker until you like the look of the areas you want darker. Don’t worry about the insect, or anything but the areas you want to adjust. This is a before and after and you should have something like the "after". An overall darker image…



Now here is where the mask comes in. Add a white mask to the “DarkBG layer and make sure the mask thumbnail is selected (highlighted)…



Choose your brush tool and set it for a soft and rather largish brush. Make sure your foreground color is set to black (which will be the color of your brush). http://docs.gimp.org/2.2/en/gimp-tools-paint.html

Set the opacity level of the brush low. I used something below 50%, then use the brush tool to start “painting” (click/hold- move the mouse) over and around the spider. You should see the area you are brushing start to lighten up. You can paint over it with repeated passes to gradually build it to where you want it and lighten as much area of the scene as you want.

Now, white reveals, black conceals.

Pure white in the mask allows the corresponding pixels on the layer to be completely visible in the image. In other words the darkened layer (DarkBG) with the white mask is visible. You took the black brush and “brushed away” the white mask and in so doing made the layer under it (lighter layer) visible where you “brushed away”.

Before and After masking/brushing...



Brian, at this point you can also adjust the opacity of the layer itself (DarkBG) to control the amount of dark/light. If you have "brushed away" to the equivalent of 100%, where that area is will not be affected by a layer opacity adjustment. Don’t be afraid to experiment. Maybe varying the brush size, opacity of the brush, etc. Go wild!

I hope this helps Brian. I’ll tell you what. I’ll stop here and you see what you think. If you already know this, then cool. If not and it helps maybe we’ll go some more and maybe look at black masks to accomplish the same thing.

Let rip any questions, comments, disparaging remarks, whatever urge strikes!

[JBW]

Hi Brian,

I understand the confusion. Layer masks are confusing at first. But like anything once you get the hang of it it’s a piece of cake. So no worries! I’ll try to take a shot here and see if I can help make things clear as mud!

Please note that I don’t use Gimp, but I’ve had a cursory look around and masks seem very similar in Photoshop. I think the issue may get even more confused getting advice about using gradient masks, shaped masks, etc. when one is first trying to get a handle on all this. I apologize in advance if I am about to oversimplify but if you are already to this point, maybe someone else isn’t.

Let’s break it down!

So you have your shot where you want it except you want to darken the right and left side. You have merged all layers except the base layer. So you should have two layers in your layer stack.

First, if you wanted to keep all those layers rather than merge to only one there is a nifty command that makes that happen. It is called “stamp visible”. Highlight (click) the top layer in the stack and then find how to “stamp visible” in Gimp (I googled it and know it can be done). What this does is merge all layers into a new layer on top of the stack. A new base layer on top. Since it is a new merged layer the other layers below it will not have any effect on the picture now. You are working with a new merged layer. The advantage is that you can always get back to your other layers if need be. The disadvantage is it increases the file size. But when you are completely done you can always go back and delete any layers you want to lessen the file size.

But since you didn’t do that we’ll go from where you are with the hidden BG layer and the merged layer on top of it.



Remember I’m in Photoshop but you should have something similar showing for your layer stack.

Duplicate the visible merged layer and rename it DarkBG or similar…



Now it is my understanding that Gimp has no “adjustment layers” and I could very well be wrong about that. So you need to use the levels tool on the duplicate layer to make the layer darker. It should make the entire layer darker but no worries. Just adjust it darker until you like the look of the areas you want darker. Don’t worry about the insect, or anything but the areas you want to adjust. This is a before and after and you should have something like the "after". An overall darker image…



Now here is where the mask comes in. Add a white mask to the “DarkBG layer and make sure the mask thumbnail is selected (highlighted)…



Choose your brush tool and set it for a soft and rather largish brush. http://docs.gimp.org/2.2/en/gimp-tools-paint.html

Set the opacity level of the brush low. I used something below 50%, then use the brush tool to start “painting” (click/hold- move the mouse) over and around the spider. You should see the area you are brushing start to lighten up. You can paint over it with repeated passes to gradually build it to where you want it and lighten as much area of the scene as you want.

Now, white reveals, black conceals.

Pure white in the mask allows the corresponding pixels on the layer to be completely visible in the image. In other words the darkened layer (DarkBG) with the white mask is visible. You took the black brush and “brushed away” the white mask and in so doing made the layer under it (lighter layer) visible where you “brushed away”.

Before and After masking/brushing...



Brian, at this point you can also adjust the opacity of the layer itself (DarkBG) to control the amount of dark/light. If you have "brushed away" to the equivalent of 100%, where that area is will not be affected by a layer opacity adjustment. Don’t be afraid to experiment. Maybe varying the brush size, opacity of the brush, etc. Go wild!

I hope this helps Brian. I’ll tell you what. I’ll stop here and you see what you think. If you already know this, then cool. If not and it helps maybe we’ll go some more and maybe look at black masks to accomplish the same thing.

Let rip any questions, comments, disparaging remarks, whatever urge strikes!

It is getting close to midnight here and i am going to give this a try but I don't imagine that there will be any good results till after I wake up. this is just what i was looking for. thanks Days and days of frustration cleared up in 5 minutes of following your instructions. This was quick and dirty but it worked and tomorrow I am going to be having fun. THANKS!

[Loose Canon]

Sleep well and Happy Painting Brian!

(Who was that Masked Man?)!

[ajohnw]

Terry has it right often about the painting. On the tutorials the selective colourisation one part way down is the same sort of thing Terry has just shown. That's here

http://www.gimp.org/tutorials/Layer_Masks/

The thing to appreciate about masks is that they allow areas to be made more transparent. I found the terminology rather odd - initialising them to white. What that means in effect is that the image the mask is attached to is completely opaque. Paint grey on it and it becomes partially transparent and the layer beneath starts to show through, the more towards black the more it will show through. The colourisation tutorial shows that well. The one above that shows how changes in the level of grey from white to black alter the transparency. He's doing his best to show general principles that's why the things he does with them are a bit odd here.

I probably shouldn't have linked to the luminosity masks but expected you to find the one above. The luminosity mask work shows how selective work can be done on a layer or layers. That might be described as stage 2 in the learning process. A lot of the tutorial is about generating these manually but the important thing is to grasp the principle. Channels can go to a selection or a mask. All good fun especially where to click. These tutorials are the best I know of as it's possible to follow all of them in the steps as he describes them. That probably is a good way to start and notice what happens to an image. Doesn't really matter what the image is.

It's really important to grasp the visible aspect and which layer or mask is being worked on. Always selected by clicking in the right place but not all that visually apparent in my view.

Adjustment layers are a bit of an odd thing in the GIMP. As I see it these are layers which do "normal" adjustments to the layer beneath. Duplicate and etc is a fairly common one. The sharpening layer isn't as common. The way I found that one was to click on the menu option to see what it did. It's very useful and I have never seen any mention of it. I assume but don't know that on PS the sharpness it introduces can be simply adjusted. Not so the GIMP so sharpness is built up by simply duplicating the layer, that's why it is very mild. There will be many other things like this buried in it. There are also numerous plugins. Taking one as just a for instance, a fake hdr look. The image this produces can also have the usual things done to it via a duplicate layer etc or the opacity of it can be adjusted to blend some of it into the original image. When ok which ever it's then just right click new from visible. Masks can also be thrown into this lot - blending certain areas only.

You might also find this tutorial handy at times and again all of the above can be added to it.

http://www.youtube.com/watch?v=FMlKVDjJFfY

At the end he just shows look what happens when I also do this - leaving something for people to play around with. There are a number of decent tutorials on youtube. What I did with this one is follow the steps and stop the video as I did them. If you find one that starts off the GIMP unlike PS hasn't got what ever often it's just best to not bother watching it. I sometimes wonder if these are planted by the Adobe mafia. Others say buy my GIMP plugin as it's better. Always demonstrated via poor use of the GIMP one.

Playing around is the only answer to learning really. That shows what the process does and an understanding of that leads to how and where it can be used.

John
-

[ajohnw]

Here's what a general pure layer set up produced Quick one that can be messed about with



That was obtained with this



Few points.

I used the original image and used the clone tool to tone down the highlights on the "onion skin" type object. Not difficult to do if you set the opacity to say 10%. That allows the change to build up slowly. It's often best to start with the largest brush size that can be used and then reduce it to tidy up. On this sort of thing it's best to stop a little short of perfection or take a lot of care. The spiders legs could do with some touching up as well in places really. The sharpening brush should do that. Rather than mess with masks or layers I feel it's best to remove problems before working with layers. Just duplicate the original image in case of a complete mess up and work on the copy. You can of course stick any other layers you might want to use on top of that.

The brightness layer can just be duplicated if needed but it's best to reduce the effect as they are added. Say 100,50,25% etc 1/2 it in other words.

Notice where I keep the undo history. Much quicker to use than the one under edit.

The contrast layer has a question mark because while it does do that it does other things as well. Unlikely to want to duplicate that as the effect is pretty strong.

I used 2 vignette layers. One is in burn mode. That's very aggressive and will saturate colours. The multiply way is more gentle. One this shot I set that to 100% opacity and the burn to about 30%. That and the other opacities can be changed to vary the result. 2 multiply could be used.

The vignette layers are generated with right click new layer - transparent. Then the gradient tool which has several options.

These layers can also be modified with a brush but go slowly to try and keep things even. If the problem areas on the spiders legs were fixed there would be no problems involved with selecting the vignette layer and painting over them with a low opacity white brush to brighten them up a little.

Another method of generating odd shaped vignettes is the same type of layer. Fill with black and then select and area of the right shape using say the oval select tool and cutting it out and then run a heavy gausian blur on the result. Various shades of grey can be used instead of black too. Even colours. Hard to predict what the result will be so a bit trial and error.

It can be interesting to click through the various layer modes when doing this sort of thing to see what happens. They tend to be grouped in sets that do similar things. Some might be more effective when an image is inverted.

John
-

[ajohnw]

Forgot to mention this.

If you start painting on things the plugins generate or some other types of work it's important to watch this which explains why using the eraser on a jpg paints white rather than making the areas transparent.

http://gimp-tutorials.net/node/200

The other way of doing this rather than duplicating the layer is to right click on the image that has been loaded and click add alpha channel. It's always a good idea to duplicate an image anyway before doing anything else.

Some things need tidying up by painting white or black or something in between. This is the other option that crops up.

John
-

[Loose Canon]

Hi Brian,

If we're on the right track then while we’re at it what about black masks?

Using the same image and two layer stack (invisible BG and merged layer) perform the same procedure as before (darkening the duplicated merged layer) until you get to the point of adding a mask. Here is the darkened image and the corresponding layer stack.





Now add a black mask to the layer “DarkBG” and make sure it is selected (highlighted). When you do you will notice the darkened layer is no longer visible. White reveals, black conceals. Pure black on the mask blocks the corresponding pixels on the layer from appearing in the image.



So choose the brush tool again, same approximate settings (soft, largish, low opacity) only this time set the foreground color to white and this will be the color of your brush. You will be painting a white brush on a black mask to "paint it away" so to speak.

Now this time instead of painting over the areas you want to lighten, paint over the areas you want to darken. The sides, top, wherever. The darkened layer starts to become visible. Just the opposite of the white mask. Again you can gradually build it up to where you want it by repeated passes with the brush.

Okay, say you got a little brush happy and made a swipe over the spider and darkened him. Oops! Didn’t want to do that. You could “undo” the brush stroke or (and this is cool) reverse the color of your brush from white to black (again, set the foreground color for this. note that it will be the same color {black} as your mask) and paint back over the spider. You are essentially painting the mask back on by doing this and the spider should lighten back up for you. You can do this with either the white or black mask to reverse what you have done.

Again, after you are finished you can further adjust the layer opacity to control the level of darkness you see.

Two different masks to accomplish the same thing in opposite ways. Once you start getting the hang of this you can choose either a white or black mask to suit the occasion and what you are looking to do.

You could use masks to do local sharpening. Deal with backgrounds as we have done. Deal with the subject. Example: you could lighten a layer looking at only the spider as you do so. Once you got him lightened up like you want, add a black mask and paint the spider back in with a white brush to lighten him (and only him). Very detailed brushing so don’t forget to zoom in tight for detailed work and use a smaller brush size to "stay in the lines". Lots of cool ways to use masks and a hugely powerful tool.

Hopefully, this will give you a couple of examples to play with to help get you going with layer masks and help illustrate basically what they are about.

Be sure to fire off any questions, comments, good jokes, as the mood dictates Brian!

[JBW]

Don’t be afraid to experiment. Maybe varying the brush size, opacity of the brush, etc. Go wild!

I hope this helps Brian. I’ll tell you what. I’ll stop here and you see what you think. If you already know this, then cool. If not and it helps maybe we’ll go some more and maybe look at black masks to accomplish the same thing.

Let rip any questions, comments, disparaging remarks, whatever urge strikes! [/QUOTE]

Here is the good news; no disparaging remarks will be forthcoming from this happy camper. I did look for the 'stamp visible' plug in but none of the pages were sill up and running. But i will keep looking. Today I got a rather nice shot of one of my friendly crickets so i used it as a model. I took your advice and played around with what I could put in the mask and how many masks I could add. The end result for today is one rather good shot (or so I think) with a gausian blur mask to highlight the cricket and a second gausian blur with a canvass mask. Not so pretty but at least now I know how to add multiple masks.

Thanks to one and all for the encouragement and the advice. The journey continues!

The only down side is that for some unknown reason when I use masks I loose my ability to add rounded corners.
original

gbm

canvas

[ajohnw]

Stamp visible is no longer available because right click new from visible in the layer window does the same thing.

I suspect you need to back step a bit. Most on here work from raw and make certain adjustments to the image at that stage. It's often a good idea to do the same with jpg's. I usually use curves but levels is easier. The idea is to get the output where it needs to be using all of the input and none of the dead space in the histogram but leaving room for post processing as I explained before. I did that with this one and also used smart sharpen with a rad of 2 due to the size of the image. May as well start with as good an image as possible.



This is the layer set up I used



After that had been done I then applied levels to the base image again. Chopped of some of the black on the input and adjusted for best contrast on the cricket. When levels are used it's the centre pointer on the input that is used to adjust it. In both cases on this one it's the effect on the cricket that counts most.

As to the fist use of levels leaving room for further adjustments personally I am rather glad Colin mentioned it and annoyed that I didn't think about it that way. I was sort of doing it that way but Colin's view of that stage is much much better - I just want to leave room to pp it further. If things run from 0 to 255 something is going to clip. On the imput side if there is no activity at either end of the histogram there is often no point in including it in the output. Setting the output at say 20 to 230 the 1st time it's used may be a suitable starting point. It all depends on the image. Copping black of the input if possible helps get rid of noise at that end. Much depends on the shot in that respect.

Curves is far more flexible than levels but it's a lot easier to mess things up so takes a lot of practice.

John
-

[JBW]

Stamp visible is no longer available because right click new from visible in the layer window does the same thing.

I suspect you need to back step a bit.


As to the fist use of levels leaving room for further adjustments personally I am rather glad Colin mentioned it and annoyed that I didn't think about it that way. I was sort of doing it that way but Colin's view of that stage is much much better - I just want to leave room to pp it further. If things run from 0 to 255 something is going to clip. On the imput side if there is no activity at either end of the histogram there is often no point in including it in the output. Setting the output at say 20 to 230 the 1st time it's used may be a suitable starting point. It all depends on the image. Copping black of the input if possible helps get rid of noise at that end. Much depends on the shot in that respect.

Curves is far more flexible than levels but it's a lot easier to mess things up so takes a lot of practice.

John
-

Your image certainly has more pop. Levels is easier and curves does mess up in a hurry. I am going to have to try to do more to the base layer. Give me some time to digest and then we shall see how it goes.

[Peeshan]

One quick and easy technic that I often use is (the idea behind is similar to HDR photography):
1. make two duplicate, one Darken Only and one Lighten only (in that order most of the time)
2. with level tool, generously push up the black on the DarkenOnly and the white on the lighten only (often to keep a crisper contrast I also reduce the grey to 0,92 or so on the lighten only layer)
3. reduce transparency of both layer, for example 80% darken only and 70% lighten only

That way it's easy to control the output and avoid burning or losing details on the darker part of the picture, or on the contrary to provoke a high contrast. then it's up to you to find the ideal combination, sometime even merge to the lighten only layer a bright screen layer if you need to boost the lightness, and so on.

[ajohnw]

Brian. One thing in respect to curves that might interest you can be downloaded here. I only use them from raw but maybe they can be loaded into the GIMP as well.

http://fotogenetic.dearingfilm.com/downloads.html

As you probably work from jpg's your camera has already used one. Most likely an s shaped one.

It may be possible to load these curves into Adobe products too.

John
-