The basics of blend modes and masking

Started by 3star, May 08, 2021, 04:51:23 PM

Previous topic - Next topic

3star

First post here, first of all I'm impressed by how supportive and useful this forum is, seems like a great community.

I've been working with Terragen for about a month developing a vector displacement workflow from Maya via Mudbox. I spent way to much time with Bryce in the before times and I have to say I very impressed with Terragen. I have watched some great tutorials from geekAtPlay and terraTuts and read the help docs on all the nodes I'm using. Still I haven't found a fundamental explanation on the blend modes applied to the main input of a surface layer. I have been working in the Maya universe for more than a decade where we have "layered textures" (textures combined using alpha and blend modes) and "layered shaders" (entire shaders combined based on the transparency of each shader in a stack, a shader with 0 transparency completely replaces any below it). I'm trying to figure out how to use masks to create the "layered shader" effect in which one shader (color and displacement) completely replaces another. I have made this work by masking both surface layers with the same mask (one inverted), but this seems like a recipe for weirdness in the gradient edges of the mask. I am guessing there must be a simpler way with a single mask, but I need to understand more about blend modes and masking in Terragen.

From what I can tell a surface layer is blended with its main input using "mix" blend mode for color but "add" blend mode for displacement (test_layerMasking_mainInput_v1.tgd), which makes sense to build up a multilayered terrain type. However I can't find a way to mask two completely different terrains, ie to blend both color and displacement in "mix" mode. I have tried everything I can think of with the mergeShader (test_layerMasking_merge_v1.tgd) node but cannot find a way apply displacement from a second surface layer using "mix" blend mode (based on the masking of the second surface layer, where masking is white then the second layer replaces upstream data) whether I mask the surface layer or use the mix controller. I am particularly confused by the way that mix controller works with "mix" blend modes, it appears to work as expected using "add" blend mode but has no effect in "mix" blend mode. 
So my questions would be: 
Is there a way to do this without using the same mask twice? 
Am I missing anything about blend modes?
How should one use mergeShader.mixController correctly?

Thanks!
Steve

Dune

Welcome, first of all. I think it's better if you'd explain what you want to achieve with the setups you share here, as they have several things that do not work. In the merge file you have linked the fake stone shader to the displacement input of a surface shader and that is not working (well, it does, but always gives extremely hard edges). Fake stones go into a child input in priciple. The simple shape mask has a very hard edge too, giving ugly hard sides to the masked area. So that should be beveled or smooth stepped by some degree.
Then the merge shader; when mixing 2 variables/lines, it's best to put them both off the main line and have the merge shader sit on another surface shader as child.
I changed this file to a bit better, and hope that helps. I'll have a look at the other one too.

Dune

Also did some changes to your second file. Perhaps this is what you want to achieve, just guessing ;) Hope it gives you some insight.

3star

Thanks for looking at this @Dune. My attempts were not meant as "real world" examples, just trying to figure out how to mask one shader over another, both color and displacement, with a single mask. Looks like using the mix controller of a merge shader is the way to do it, it wasn't working for me because I had "mix to A = 1", which apparently overrules the mix controller, as does 0, at 0.5 it works. thanks for the tips on when to use child layers. I got it working as expected. Once I have some landscape finished I'll post some WIP images. The second version you uploaded is useful too as a "real world" example.

Dune

You're welcome, and I'm looking forward to your WIP's.