Masking Tutorial

[Stormlord]

Masking Tutorial

This small tutorial is about masking and it will come in two parts.
It's about how to mask, terrain and/or specific details like water, plants ect. in a scene?
How to, tips and tricks, ect.

As bonus I attach the example scenes developed for this tutorial...


Example Scene by Stormlord
Masking example

Note:
The scene itself is the Terragen standard scene, which pops up if you start Terragen.
I just have set different cameras at some prominent spots.
A mountain (frontal view), a valley and a mountain ridge.
The positions are carefully and specially selected for test purposes, like vector masking or developing basic shader setup in mountainous terrain.
Just try out the different cameras and see yourself.
It's a good starter scene!


The situation
Often you have a nice Terragen rendering, but details in the scene needs more color or details have to be more brightened.
So the problem is, how do you mask only the details of interest? E.g. the water surface only or the single tree in the foreground?

For these problems I have created a standard procedure to mask out specific details.
I just render everything in black. Except the objects or details which are of interest, they are rendered in white.

So basically you need a white and a black material.
When masking water surfaces in black, reflections has to be set to zero !


White, gray and black
I found out, best is a whitish material with illumination set to 3.0 as white.
An illumination material is better than a white one, because this way you can deactivate all lights in the scene.
This speeds up the render time.

When you're rendering the detail with just a simple white material, you will end up with gray tones in the mask.
So I set illumination to 3.0. This gives you a bright white output, instead of a whitish gray.
Grey tones in a mask result in an unsharp mask.

After a lot of masking, I found out, that illumination set to 3.0 is mostly optimal!
If you go up to 100 for example, then the contours wear out and become unsharp.
A value below 3.0 will sometimes end in white-grayish colors, which is sub-optimal.
So it is best to stay with 3.0 to get a clean, sharp and contoured mask for proper masking.


The example scene


Example Scene by Stormlord
(Untouched rendering)

This is my example scene based upon the "Standard Scene" in Terragen.
This scene will be used to develop basic masking scenes as an example.


Masking terrain


Masking plants (here grass clumps)


In both masking examples, I used a white material with illumination set to 3.0 for masking.

The water has a standard black material. The reflection in the water shader is set to 0.0 (no reflection).
Atmosphere is deactivated, so the sky will not be rendered and remains black.
All lights are deactivated, which results in faster render times.


MASKING PROCESS
How to do...

1. At the end of the Shader Node set a white luminous material.
2. Water surfaces get a black material with NO REFLECTION !
3. Disable Atmosphere Node from the planet (no atmo = black)
4. Set in objects like plants inside the parts shader "Luminosity" to 3.0.
Values bigger than 3.0 wear out the contours.
Values below 3.0 can sometimes end up into more gray colors.
Stay with 3.0 !
5. Deaktivate all light sources.
6. Depending on the mask, set color to black or white.

STORMLORD

Creating all masks in Adobe Photoshop CS 5.5

After rendering all necessary masks for further manipulation, I put them all together.

Here is how to get the mask correctly in Photoshop:
Pull the rendered mask in and lay it on top of all layers.
Go inside the mask channel tab, hold left STRG, and click with the left mouse button on the RGB channel.
So all white pixels will be selected.
Then you can add or substract other areas to get the desired mask. Mostly, terrain, water, sky, plants and objects of special interest.


Screenshot Adobe Photoshop CS 5.5
(All masks ready to go)


Manpulation with masks


Example Scene by Stormlord
(Optimized)


Example Scene by Stormlord
(Direct comparison side by side)

Everything is possible with masks (you just need to create them)!


Example Scene by Stormlord
(Planetarium style)


Example Scene by Stormlord
(All layers in Adobe Photoshop CS 5.5)


Real life example

There is the famous Terragen 4.0 Benchmark which everybody can download here...
https://planetside.co.uk/terragen-4-benchmark/


Terragen 4.0 Benchmark
(Rendered as desktop in 4K and optimized with masks)


Terragen 4.0 Benchmark
(Direct comparison original/optimized)

End of part 1
These are the basics about masking in Terragen.

The second part will be about using the Layer Manger and very usefull tips and tricks.
So stay tuned...

STORMLORD

[Stormlord]

Masking Tutorial - Part 2

Here are two powerfull tricks...
1. Add a Color Adjust Node after a mask and swap the Black and White Point to invert it.
2. Lower the Gamma in the inverted mask to refine it.


Masking Trick 1
Inverting a mask by swapping white and black point in a Color Adjust Node.


Masking Trick 2
Gamma set to very low values refining a mask


Render Layer

For those who have a license for Terragen professional it is possible to render seperate elements with the Render Layer Node.
There are some elements which can be very usefull as masks for further editing.


Terragen 4.7.15
Render Layer / Render Elements Tab


Mostly relevant for masking are these layers:
- Alpha
- Surface Alpha
- Cloud Alpha
- Atmosphere Alpha
- tgSurfDepth

You can save the output as .exr or .tif file (in 32,16 or 8-Bit depth).
I use 32-bit for .exr and 16-bit for .tif files for maximum quality.


General advice

Please always save your rendering at least as 16-Bit .tif file (or 32-bit .exr).
Further manipulation can lead to unwanted color steps, if you save your image only in 8-Bit.
Then most problems occur later in postwork in the sky or water, where you have very fine color steps between pixels.
Same with saving... save them not as .jpg file if you want to have your rendering in a very good quality.
Better save them as .tif or as .bmp (for desktops). This guarantees you a very pleasant viewer experience.


Render Layer at a glance

These are the render elements from my example scene.
Please pay attention to render element tgSurfDept on page 2.


Render elements 1


Render elements 2
Special Case "tgSurfDepth"

While tgSurfDepth is saved in 16-bit as tif file, you mostly see only a black rendering.
This is because the differences here are so subtle. So let's try auto contrast!


tgSurfDepth


tgSurfDepth with auto contrast

This way you can use the mask with auto contrast tgSurfDepth very well as Z-Depth!

So now this is the end of Part 2 of this tutorial.
If you have some more cool masking tips and tricks which could be added here, so feel freee to post them.

At the end and as a Bonus, I made also PDF for you, which you can also download.
Download Masking Tutorial PDF

STORMLORD

[Stormlord]

Many people render their images and the vegetation inside their scene comes out to dark.
So why not lighten it up to earn better results?

I often render a vegetation pass and overlay it with negative multiply to light up my vegetation.
Please allow me to explain it, how you can do it also. You will have a greet benefit from it!

I prepared a small sample scene which has the OldPine.tgo embedded, I'll attach all files to give you a good start.
So now, here's how to...


1. Import your vegetation in your scene, in this example it's the oldbark.tgo
Normally, you would render your scene here and get very dark (to dark!!!) vegetation as usual

I normally lighten the vegetation up this way...


2. Set the light that it will come somewhere from the FRONT!
No matter where your sun is currently set in your scene, at the end we apply a vegetation pass to it, so the light for it has to be set, that it comes from the front to get a bright render.


3. Inside the Parts Shader, set all Base Colors to 1.0 (instead of standard 0.5, this will brighten up the textures)


4. This is how you could render your scene, but wait.. there's a much better way!


5 - Render a vegetation pass instead (you will be more flexible with it)
Set at the end of the Shader Network a black shader, detach the Atmosphere (= black), set water to black with no reflection


6 - In Photoshop I lay above my render the vegetation pass with NEGATIVE MULTIPLY, then fading it between 0-100% the way it comes out best

That's the way I do it.
And you can do it now as well!

STORMLORD

[schmeerlap]

In your lighting up vegetation tutorial will there not be a discrepancy of original sunlight direction and shadows on terrain, with the adjusted sunlight direction on the tree render pass, which won't match with the tree shadows on the terrain? It seems your tutorial is OK as long as you don't change the direction of the sun, which will always have to be from behind to give good lighting on the trees.

[Stormlord]

I will give a more in depth example, John.
Actually, you will stay with the rendered image, but illuminate darker areas.
By fading the vegetation pass into the rendering with negative multiply, it will light up darker zones and brightens up the rendered vegetation.


Pacific Island 2024
Original rendering with some adjustment


Pacific Island 2024
Overall Negative Multiply 35% only in darker areas

How to get this mask for the darker areas of the original rendering...?
-> Copy the Layer, make it Black and White, Set Threshold value (as you intend to get the mask), apply a Gauss Blur (voila, only darker areas are then masked)


Threshold Value applied to a screenshot image

 
Pacific Island 2024
Vegetation Pass (Trees and Grass with 35% Negative Multiply


Pacific Island 2024
Vegetation Pass (Trees only) with 35% Negative Multiply
-> Center Palm Core only

Please allow me to mention, that I always render several masks to manipulate exactly the areas of interest!
In the next picture, I show used masks to manipulate only the areas of interest.


Pacific Island 2024
Vegetation Passes

The glittering on the water is rendered with the Sunlight Degree + 180° -> from the direct opposite direction of the Sunlight 01 in the scenery.
Render Layer -> tgSurfDirectSpec -> Overlayed with 2 Layers with Negative Multiply 100% (See Screenshot Adobe Photoshop)


Pacific Island 2024
tgSurfDirectSpec

I hope, that it is more clear now, how to pimp up your next Island

STORMLORD

[schmeerlap]

Thanks Dirk
I get the Photoshop adjustments. It was just your moving of sunlight direction in Terragen that foxed me.