First post here, first of all I'm impressed by how supportive and useful this forum is, seems like a great community.
I've been working with Terragen for about a month developing a vector displacement workflow from Maya via Mudbox. I spent way to much time with Bryce in the before times and I have to say I very impressed with Terragen. I have watched some great tutorials from geekAtPlay and terraTuts and read the help docs on all the nodes I'm using. Still I haven't found a fundamental explanation on the blend modes applied to the main input of a surface layer. I have been working in the Maya universe for more than a decade where we have "layered textures" (textures combined using alpha and blend modes) and "layered shaders" (entire shaders combined based on the transparency of each shader in a stack, a shader with 0 transparency completely replaces any below it). I'm trying to figure out how to use masks to create the "layered shader" effect in which one shader (color and displacement) completely replaces another. I have made this work by masking both surface layers with the same mask (one inverted), but this seems like a recipe for weirdness in the gradient edges of the mask. I am guessing there must be a simpler way with a single mask, but I need to understand more about blend modes and masking in Terragen.
From what I can tell a surface layer is blended with its main input using "mix" blend mode for color but "add" blend mode for displacement (test_layerMasking_mainInput_v1.tgd), which makes sense to build up a multilayered terrain type. However I can't find a way to mask two completely different terrains, ie to blend both color and displacement in "mix" mode. I have tried everything I can think of with the mergeShader (test_layerMasking_merge_v1.tgd) node but cannot find a way apply displacement from a second surface layer using "mix" blend mode (based on the masking of the second surface layer, where masking is white then the second layer replaces upstream data) whether I mask the surface layer or use the mix controller. I am particularly confused by the way that mix controller works with "mix" blend modes, it appears to work as expected using "add" blend mode but has no effect in "mix" blend mode.
So my questions would be:
Is there a way to do this without using the same mask twice?
Am I missing anything about blend modes?
How should one use mergeShader.mixController correctly?
Thanks!
Steve