If you initially assign your population density to an image alpha type mask, could you then use that mask with broadened edges for your texture mask.
Like, a crisp line for the trees, then second mask image for texture widens that line, overpainting original mask?
I suppose this would depend greatly on the size of the area you are populating.
I remember other programs you could create a mask right on your terrain bmp, I think you could do this with a ratio too, where if you just wanted one tree or whatever, you got one tree per dot. (Or I could be hallucinating)