adding rendered trees over a photo?

[Dune]

I have to add rendered trees to a photo. As I've never done any comping like this (and never used render elements), how should I go about it in TG and Photoshop? I need the trees and a mask to eliminate any TG background and atmo, and paste just the leaves/trunk over the photo. So I will probably need a certain render layer. But which one? 
I also need the shadows without any ground, separately I guess. That can probably just be multiplied over the photo in photoshop.
Can anyone enlighten me?

[Kevin Kipper]

Hi Dune,
Great question.  First let me point you to the online documentation for compositing Terragen render elements.  In this wiki, we used Fusion as our compositing software, but you can substitute Photoshop for it along with the appropriate Photoshop blending mode: https://planetside.co.uk/wiki/index.php?title=Compositing_Terragen_Render_Elements

To help you understand "all" the render elements, think of it like this.  A normal render from Terragen outputs a "beauty pass"; that is, all the terrain, objects, atmosphere, lighting, etc. are taken into account and create the final image.  Render layers allow the user to assemble these separate element back into the beauty pass. 

The Render Layer window is divided into 3 sections.  Each section allows the user to break down the render layers into more specific elements.  For example, to recreate the beauty pass from Terragen, all you really need to start with is the SurfaceRGB, CloudRGB, and AtmosphereRGB render elements.  If your combine these three render elements on top of each other in an additive mode, you'll get the beauty pass.  This allows the user to modify each render element as they see fit.  Perhaps changing the colour saturation of the SurfaceRGB element, or increasing the brightness of the cloud element, etc.

You can go deeper too, and that's what the second section is for.  Here you can break apart direct lighting verses indirect lighting.  All you need to do to recreate the "full" lighting is combine the two elements back together in additive mode.

The third section, the Data section, allows you to export other types of data such as zdepth, etc.

So for your project, think about what you might want to have control over in Photoshop.

My approach would be something like this.  I would save a tiff file from the renderer's Output Image File param.  This would give me a beauty pass with all the Terragen tonemapping and how I expect the final composited render element to look.  If I thought I would only need to breakdown the render into "basic" parts, on the Render Layer's Render Elements tab I would enable the checkboxes in the first section.  If your Terragen project has no clouds in it, you don't need to enable CloudRGB and CloudAlpha.

If you want to render your trees against a black background, group the tree objects together in Terragen, then assign the group to one of the Object groups under the Render Layer's Object tab.  Set the group's render mode to Visible and enable Cast Shadows And Other Rays.  Set the All Other Objects render mode to invisible, but you may need to enable their shadow casting and other rays, if they have an effect on the trees.  For example, the buildings in the photo casting shadows on the trees.

One last note, if saving render layers in EXR format, Terragen's tonemapping will not be embedded into the EXR images.  That's why I save the output image, in order to have a reference to how I want it to look. 

I would be curious to watch the progress of your shot on the forum.  Don't hesitate to ask questions, we're happy to help!

[Dune]

Great. Thanks very much Kevin! I'll do some experimenting and follow up on this here, once the commission is official.

[pokoy]

Kevin's suggestions are great, however they might be a bit too cumbersome or inpractical with Photoshop. The 'problem' with EXRs and Render elements would be that they need to be composed in linear float 32bits which Photoshop is not really good at. Photoshop also doesn't support de-multiplying alpha so you'll have to tweak the alpha channel a bit.

Also, most production renderers mostly have dedicated background/matte materials that do not display in the render/alpha or let you specify how they look to the direct visibility rays and other ray elements like reflection, GI etc. Without it you need to get around that.

For the trees, I'd just save the final output as Tif (so it's gamma corrected plus has all the tonemapping baked in). For the alpha channel you'll need to disable direct visibility for anything else.
For the shadows, I'd simply render the trees on a white ground (plane/terrain) and just multiply it over your background image in Photoshop. You would probably need to do some level correction on this render because tonemapping will probably move your whitepoint to a non-white color. Or, if TG supports a shadow element, use this one. This will probably be linear so you'll need to apply a gamma correction on it.
If the ground is not flat you'll need to re-model the ground so the shadows 'follow' whatever the geometry is.

[Dune]

Thanks very much for additional info. I don't really know what demultiplying alpha means, but I guess I'll find out once I try this.
What do you mean by 'For the alpha channel you'll need to disable direct visibility for anything else'? You mean in TG's render elements settings?
I was also thinking of rendering twice; trees and shrub just floating (atmo set to secondary only) so I can make a transparent tif and easily paste over in PS, another render for shadows with trees set to invisible, indeed on a white ground and multiply over in PS.
I also thought about building the undulating ground, but I think that won't be necessary.

[pokoy]

'For the alpha channel you'll need to disable direct visibility for anything else' - what I meant that whatever is behind the trees doesn't contribute to alpha, in case there's a terrain or something else behind it.

Yeah, in PS this 2 layer approach is quick and effective as it gives you good control for blending and correcting layers individually, and it's an easy setup here.

In my view, TG lacks quite a bit functionality in terms of composing even if you have a fully fledged comp app. No additional arbitrary mask buffers (so you could add only a few selected objects or items to a mask element), no matte/shadow shaders etc. Even if you went the render element way you'd probably end up needing to render different visibility setups forcing you to keep track what's enabled or disabled for each render setup etc. To be fair, this is sometimes needed even with all the options available.

[Dune]

Thanks for the quick reply. Going to work on it...

[Dune]

I can probably keep it very simple. 
1: for only shadows set all tree objects to invisible, and render just planet (base color white) and atmo (whitened, haze raised), or a white background card. Multiply in PS.
2: render only the trees with all atmo on, but not rendering planet or atmo, resulting in floating trees on black. 

The only thing then is that I need to cut out the trees by using the alpha. I don't think TG can export/save a transparent tif/exr? If I save the render as exr and open in PS as transparency, the black background is still there. So is there any way to really have only the trees?

[pokoy]

Not sure, but it's a simple thing to do in PS. Or is there a reason you'd want the file already to be processed that way where pixels where alpha is black are already removed?
If TG saves to PNG you might try that one.

As for EXRs in PS - PS just doesn't support EXRs correctly, just expect it to do strange or unexpected things with it

EDIT - this looks nice already! Sure, the perspective is slightly off

[Matt]

You might find this useful for generating a shadow pass:

https://planetside.co.uk/wiki/index.php?title=Shadow_Catcher_Shader

Alpha channels aren't included in TG renders in the normal way, but you can save the alpha to a separate image. One way to do this is to simply hit 'A' on the keyboard in the Render View and then save that image. Alphas can also be saved as Render Elements / Extra Output Images if you want to go that route.

[digitalguru]

If you did want to explore rendering out from TGD as Exr with render passes to Photoshop, you can try Exr-Io (https://www.exr-io.com/) it's a free download to import such files into Photoshop. It can handle channels and import them as layers i.e render elements (diffuse, specular etc) Seems later verisons of TGD can output the render elements in one Exr file, not tested as I'm not up to date with Terragen, but as a basic Exr importer it works very well.

[Dune]

Thanks guys! I did find a way to extract a transparent image from beauty pass and alpha in PS, but just wondered if a direct transparency would be possible from TG. Saves a process.
And indeed, I don't really like exr, and never save as such (always tif). Yeah, perspective could be better
I'll check out the exr-io, and the shadow catcher, thanks! 

[Dune]

The most difficult thing is to get the right angle, and POV. The focal length from the photo given to me didn't exactly match the angle I eyeballed, but I'm getting somewhere.
I still need to make the proper pathway (this was just an idea, which they need to approve), and some more stuff needed.

[Kevin Kipper]

Dune, I think you mentioned that you had an older copy of Lightwave.  It has a feature called "match perspective" that might help solve the camera position for you.  You could export the camera solve as an FBX, or perhaps just type the coordinates from Lightwave into Terragen.  Here's a video I found online about the feature: https://www.youtube.com/watch?v=wCZo1oDE7zo

[Dune]

Thanks Kevin. Very interesting feature. But I'm afraid my LW is ancient (2015), and I don't think it's in there. I'll check it out anyway. 

Perhaps if I use a card in front of the camera in TG, I can project the image, and flip between world and card to match camera in TG by eyeballing, you've just given me that idea. With a glass shader, half transparent and RTP it may even work better. Just a theory ...