Henry/njeneb hits the hammer on the nail. Some masks could also have been done with the painted shader, but I prefer to work on one PSD layered file with all the masks above each other. It is then easy to draw a textured soft white along e.g. the waterline for a reeds mask. Perhaps selecting the water in the layer beneath and filling the reed layer within that selection with black again if you don't want the reed into the water. Things like that. Do some blobs here for trees.....
I then copy the layer, make a new file and paste, reduce size until it still shows the amount of detail needed. And then a very small pure black and white bitmap of a few kB's would be my favourite for 'undetailed' masks, but bitmaps are not swallowed. So it's 8bit greyscale.