Actually masking is not that difficult. If you have a terrain of a certain ratio, just make the mask the same ratio (paint in PS over an aerial photo or so, greyscale, safe as TIF or so). Make it smaller if less detail is needed (distribution of trees e.g.), but keep the ratio. Make an image map shader and import the mask, set to Y, fill in the size of the terrain, and use as a blend shader for whatever.
You can put several masks next to each other or over each other, as long as you find out the coordinates and ratio's. Use them to displace certain parts of terrain (before compute terrain, or for minor displacement in the shader department), or use it to color areas. Hope this helps.
---Dune