PNG will work OK too. It's the jpeg compression artifacts that cause problems with jpg image masks. LZW compressed TIFFs work fin and for masks are pretty small files as well.
In some cases you may want to leave Smooth Interpolation on as it will smooth out the blockiness of a low resolution image. You can then increase the sharpness of the mask by using a colour adjust node with identical values for black/white limits. This is quite a useful technique when you're pushing the resolution limitations of a mask image.
I'll have a play with your TGD tonight.
[edit] Here, try these files. Converted the JPG masks to TIFF (they're smaller with LZW compression). I split the simple shapes out of the direct connection to the compute terrain node to use them for both displacement functions and masks. I prefer to use the blending shader rather than coverage for masking like this and and restructured to remove some of the redundant masking. Hope this makes sense (and is what you were trying to achieve
).[/edit]