The tricky part is firstly to describe the distribution of each species and then create a mask for that...
- elevation
- degree of slope
- heading of slope (e.g. sheltered from sun = wetter = higher density/higher elevation)
- gullies or ridges...
Then you need to remember that your mask quantifies the density and not the placement. In areas where species overlap you need overlapping masks with a reduced density.
To give a basic example, let's say species 1 has an minimum elevation restriction of 1000m with a fuzzy zone of 200m, and species 2 occurs below species 1. Rather than try and be clever with a second elevation restriction for species 2, you could simply subtract the mask for species 1 from the mask for species 2, providing a transition in density between the 2 populations.