Planetside Software Forums

General => Terragen Discussion => Topic started by: cyphyr on March 26, 2012, 09:10:22 AM

Title: Detail in Deep Shadows (again)
Post by: cyphyr on March 26, 2012, 09:10:22 AM
Hi gang

I'm working on a scene of a deep canyon completely covered in tropical vegetation. The trouble I'm getting is that because the canyon is so deep (2km) large areas will always be in some form of shadow. I have tried a variety of ways to get rid of the flatness of the shadowed areas. The one that offers the best results so far is to have the GI prepass padding set to just 1 (GI relative detail = 3, GI sample quality = 2). This gives enough local detail about each tree/bush to remove their otherwise flat lighting. The trouble is that this also produces a fringe about each bucket section (look at the picture to see what I mean, horizontal and vertical dark lines)). I have tried many other variates of settings including multiple fill lights (up to 8), GI+AO and none produce the "look" I'm aiming for as well (its still got some way to go) as having a high GI setting and very low prepass.

FILE NAMING :~ this is version deepGreen10. The following numbers refer to the GI settings, so the first image, deepGreen10.3.2.1.jpg, has a GI relative detail of 3, GI sample quality of 2 and GI prepass padding of 1 and I'm using Ambient Occlusion instead of GI

So any bright ideas, this has stumped me for a week now.

Cheers

Richard

ps: the top of the two images below is the one thats nearer where I want to go but it has the bucket "fringes"
pps: bridge models are just standins
ppps: added another version, just raised the prepass to 2. Result the fringing is slightly less but so is the shadow detail

Title: Re: Detail in Deep Shadows (again)
Post by: TheBadger on March 26, 2012, 10:47:53 AM
You know I cant help.
But I can tell you that this is a really cool shot! Its easy to tell that when you workout the technical issues, its going to look great. Please share the final image when your done if you are free to.
Title: Re: Detail in Deep Shadows (again)
Post by: FlynnAD on March 26, 2012, 12:06:21 PM
What does the file look like if you take your 2nd version, the one with the overly-dark shadows, save it as an exr (32-bit), and increase the shadow curves in Photoshop? Does the detail come back (i.e. is the detail in the original render but simply too dark in the tg2 exposed rendering)?
Title: Re: Detail in Deep Shadows (again)
Post by: choronr on March 26, 2012, 12:32:44 PM
I sometimes find that in certain situations, increasing haze density alone may brighten the objects and terrain.
Title: Re: Detail in Deep Shadows (again)
Post by: cyphyr on March 26, 2012, 12:35:47 PM
I've not had much luck with exr's, they always seem to come out grainy, and its a little difficult to tell at the moment as this is only rendered at 0.6 detail and 3 AA so grainy is kinda built in.

Hmm just tried it on a new version. It brings out the detail ok (needs al lot more serious work) but also brings back the bucket fringing.

|If this was just for a still then I'd just whack up the settings and go away for a long weekend. Its not though, its going to be animated over about 1000 frames (details still to be decided but it is animated) so I'm trying to get over the issues that will arise earlier rather than later. Render time is about an hour but that will have to drom for the final (or at very least stay near the same once I start bringing the detail and AA quality settings up).

Cheers

Richard
Title: Re: Detail in Deep Shadows (again)
Post by: Hetzen on March 26, 2012, 03:30:16 PM
That's a horrible scene to light Richard. The sun is infront of the camera, and there's some hard bounces for the GI to calculate down that raveen. Funnily enough I prefer the middle render.

Haze density and glow power can lighten shadows, although I'm not sure how helpfull it will be in this scene. EXR will certainly help stretch shadow detail with a curves effect. But this might just be an exposure setting if you don't want to use EXR. Otherwise maybe a low lumen fill light behind and off centre, and another stronger one towards the sun.

Good luck, because you'll definately get GI flicker when you render those 1000 frames without some artificial help.
Title: Re: Detail in Deep Shadows (again)
Post by: cyphyr on March 26, 2012, 03:48:37 PM
Quote from: Hetzen on March 26, 2012, 03:30:16 PM
That's a horrible scene to light Richard. ...

Yes it sure is and it get worse! Since the sun is so high a lot of the shadows will be cast from outside the camera FoV I'm going to have the ray detail region padding set high. Already tried on 1 and it took sooo much longer to render, gave up, but it will be necessary for animation. I'd use fill lights to avoid GI flicker but they just don't cut it with the shadow detail.

So if I don't use GI/AO I get horrible shadow detail, everything is "flat"
If I do use GI/AO then it only seems to work well with a very low GI blur radius (1, 2, 3) which for some reason gives these "fringed" buckets. At default 20 it offers no improvement over fill lighting. And of course if I have a low GI blur radius I'm practically guaranteed GFI flicker.

The animation will have the camera turning 180 degrees to look behind it so the lighting levels will change from full shadow to fully lit.

Ah well I like a challenge!

Cheers

Richard
Title: Re: Detail in Deep Shadows (again)
Post by: Hetzen on March 26, 2012, 04:50:46 PM
You're right, but tbh the second set up in your initial post, I think would work better.

Have you rendered what it looks like at 180 degrees?
Title: Re: Detail in Deep Shadows (again)
Post by: cyphyr on March 26, 2012, 05:01:45 PM
Not yet. I'm not anticipating the same shadow problems although the fully lit trees may also be overly uniform (flat) in their shading. I'll give it a go though, good idea.
Some of the best lighting I've had so far has been with the sun almost directly overhead, this way all the trees have a more rounded, less flat appearance (although looking either straight down or up may produce similar issues) and I wont get so many of those nice volumetric shadows. None of this is in any way finalised, it'll all change I'm sure before I even start animating I expect. This is more to figure out some solutions to problems I was expecting to occur, the unexpected stuff is still to come! ;D
Cheers
Richard
Title: Re: Detail in Deep Shadows (again)
Post by: Hetzen on March 26, 2012, 05:31:42 PM
I'd be tempted to keep contrast as high as possible Richard. Your foliage, god rays, and a good tracked lens flare mask will give you highlights in those shadows that will help keep interest over that length of animation. It might be useful to see where your scene is going over those 40 seconds. Maybe run a half pixel sized animation every 50th frame over night to spot any problems.

Title: Re: Detail in Deep Shadows (again)
Post by: Kadri on March 26, 2012, 05:57:18 PM

I hear that the new Alpha TG2 has some new things . Do you use it Richard ?
If no ask Planetside. Maybe it might help.
Other then this did you tried different bucket sizes?

I would definitely try it with fill lights. In animation GI makes it very hard to use.
Use higher luminosity surface settings so you get less problematic dark places.
With this you will get faster renders and if you can put EXR outputs with these settings you could get away maybe.
Edit: You could use more lights-Suns with or without shadows to bring details out. No need for real world accuracy you know.
Title: Re: Detail in Deep Shadows (again)
Post by: TheBadger on March 26, 2012, 06:36:17 PM
Also remember that when you animate, the motion blur will help to smooth out the lighting a bit. To a certain degree anyway, not including flicker.
Title: Re: Detail in Deep Shadows (again)
Post by: Hetzen on March 26, 2012, 06:38:16 PM
You're not going to get much motion blur over 40 seconds.
Title: Re: Detail in Deep Shadows (again)
Post by: Kadri on March 26, 2012, 06:41:02 PM
Quote from: Hetzen on March 26, 2012, 06:38:16 PM
You're not going to get much motion blur over 40 seconds. ;)

Why Hetzen? Is not only the time but the speed of the camera important too (from frame to frame)?
Title: Re: Detail in Deep Shadows (again)
Post by: cyphyr on March 26, 2012, 06:56:53 PM
There will be some, the camera will be turning a full 180 in a relatively cramped place.
Richard
Title: Re: Detail in Deep Shadows (again)
Post by: Dune on March 27, 2012, 03:35:10 AM
Is it so bad to have quite dark areas in a (fast?) moving sequence? I preferred the second shot as well. Kind of mysterious. You won't have time to take up all details anyway, and usually your attention is focused to contrasting areas.
Maybe lighten up the leaves of the trees just a fraction (luminosity of 0.1) to fake light/dark, blended by a distance shader?
Title: Re: Detail in Deep Shadows (again)
Post by: Hetzen on March 27, 2012, 05:42:15 AM
Quote from: Kadri on March 26, 2012, 06:41:02 PM
Quote from: Hetzen on March 26, 2012, 06:38:16 PM
You're not going to get much motion blur over 40 seconds. ;)

Why Hetzen? Is not only the time but the speed of the camera important too (from frame to frame)?

Of course I don't know the animation details, I just know that 40 seconds on one shot is a long time, so I assumed the animation would be reasonably slow . And if you have GI problems, it won't really matter what speed the camera is moving, motion blur won't solve the issue.
Title: Re: Detail in Deep Shadows (again)
Post by: Vyacheslav on March 28, 2012, 07:50:07 PM
Not standard light.The Plan and sphere.
Title: Re: Detail in Deep Shadows (again)
Post by: Cyber-Angel on March 29, 2012, 12:59:04 AM
If that canyon where real then I'd say that the lighting you have would be about right. There would be no way that real sunlight would give full illumination down there, given canyon wall height, and density of the vegetation down there which would absorb, rather than reflect light.

Regards to you.

Cybe-Angel
Title: Re: Detail in Deep Shadows (again)
Post by: coremelt on March 29, 2012, 05:25:56 AM
pro compositor weighing in here.  This is the sort of issue you really want to solve in compositing.  Make sure that the details are there in the blacks by taking a render into photoshop and pushing the gamma up and down.  As long as it's not clipped solid black you're good.

Render it out as is and then tweak your RGB curve in After Effects to pull the detail in the blacks up.  You might fiddle with this for days in TG2, it can be solved in 1 hour in comp.

I'd either render 16 bit tiff or EXR and then fix it in comp to get whatever grade look you want.
Title: Re: Detail in Deep Shadows (again)
Post by: Matt on March 29, 2012, 05:40:47 AM
Hi Richard,

1) Did you mean "GI blur radius"? A low value like 1 is likely to give problems at bucket boundaries like that. Perhaps you can increase it by a small amount until the border problems are gone.

2) You can eliminate GI flickering with GI cache files in v2.4. Generate a cache on every 20th frame or so (guessing from what you've said about the shot), set the number of files to blend to 4 or 5 and you should be fine.

Matt
Title: Re: Detail in Deep Shadows (again)
Post by: cyphyr on March 29, 2012, 05:51:20 AM
Thanks guys

Yes I do think this is going to be most easily sorted in post comp. Its a little difficult effectively working blind. Render something and then drop into PS, rinse and repeat with altered settings! One of the settings I altered recently made the last bucket render far too long (I gave up after a day!). I'm changing both terrain, shaders and lights to get the final "look" just right.

Quote from: Vyacheslav on March 28, 2012, 07:50:07 PM
Not standard light.The Plan and sphere.
Can you expand a little on what your doing in these scenes. I don't understand the sphere in scene 3 ... what dose the Strata shader plugged into the opacity channel of its default shader achieve?

Coremelt. Have you had much success in using 16 bit tiff or EXR from TG in animation. I've always found the extra channel "width" ultimately led to more noise, are you using particularly high settings?

Quote from: Matt on March 29, 2012, 05:40:47 AM
Did you mean "GI blur radius"? A low value like 1 is likely to give problems at bucket boundaries like that. Perhaps you can increase it by a small amount until the border problems are gone.

I was using such a small GI blur radius as this was the only way I could get the subtlety of lighting in the "in-shadow" trees. Anything higher and they seem to loose their "round-ness" and look like flat bill boards. I wanted just a little shading to give the trees some shape. Slightly darker within the volume of the tree and beneath it.

Animation wise, yes the 2.4 features will prove invaluable (indeed I doubt the project would be possible with out it!)

Cheers

Richard
Title: Re: Detail in Deep Shadows (again)
Post by: coremelt on March 29, 2012, 05:55:52 AM
not yet because we're only just doing our first job using TG right now.  I'll let you know what happens when we get to that stage.

Noise in the blacks can also be dealt with in comp pretty easily using a median filter only on low luminance areas or a temporal filter again only on the blacks.  From a compositors point of view, we always love having more color depth whenever we can and we can usually get something useful from it.
Title: Re: Detail in Deep Shadows (again)
Post by: Oshyan on April 01, 2012, 12:23:54 AM
I think you ought to be able to get the same/similar/perhaps even better result in shadow detail by increasing GI settings (relative detail most likely). I understand why reducing blur radius helps, but it's clearly impractical to have it below a certain limit. So reduce it until just before you see tile boundaries, then increase relative detail (and try playing with sample quality too perhaps) until you get the detail you want. May not work, but worth trying since you haven't cracked the basic scene lighting setup yet.

All that being said I found the middle image to be most appealing, even though it was darkest and, I gather, not what you want. It seemed much more realistic than the other two, which both seemed artificially lit in the shadows, sort of HDR-esque.

As others have said, I think dealing with this in post is going to be the most expedient and effective option. The trick will be to get the quality high enough to give some info there in the shadows to work with, and to reduce noise as much as possible so when you pull the shadows up, it won't get too grainy.

GI Caching should take care of any flicker, with the right settings. It's not necessarily totally intuitive, but as Matt suggests, rendering cache files every x frames (e.g. 10 or 20) rather than every single frame and then blending lots of frames (e.g. 5) should work. It basically makes sure the GI is averaged over a larger time scale.

- Oshyan
Title: Re: Detail in Deep Shadows (again)
Post by: cyphyr on April 01, 2012, 08:27:36 AM
Well here's the latest attempt. I'm getting the colour/tone depth I'm after but the render tomes are too extreme to be a candidate for animating. Population takes over two hours to populate and the frames are taking over 6 hours (err thats 250 days of animating!). I don't really see where I can cut any further corners without seriously compromising quality. And there is a lot of work on scene construction still to go buildings, bridges, hanging vines etc much of which will be casting volumetric shadows which will also add to render time.
The two images below are unprocessed (apart from a linear conversion to 8 bit).
I think, unless someone can come up with a miracle solution, this will have to be consigned to just being a still image.
Thanks for your ideas.
Cheers
Richard
Title: Re: Detail in Deep Shadows (again)
Post by: Kadri on April 01, 2012, 02:35:37 PM

It is up to your scene and what kind of animation path you use etc. Richard but if it takes so long i would definitely use a fill light setting (i know we said it already) and make use of billboard objects (possibly 4-5 ones grouped together). I would make 3-4 different billboard object grubs (this would bring the population time down to 20 minutes or so maybe) and put them a little on the far side of the animation path. Not neccesery maybe but you could use low quality object  mixed between these billboard object to make them look more 3D like. I doubt that anyone could spot everything in an animation. . You could get away with more then you think.

Not one guy did say why did you use only 1 object in this animation until now! From now on it doesn't count guys :)

http://forums.planetside.co.uk/index.php?topic=14208.0

Yes only 1 object.

For example the far side of the canyon you could use (maybe you have) little green colored rough displaced terrain only and nobody would know it .
For animation you have to cut corners. I think you have still the "perfect still image" mentality Richard  ;)

And Richard you know TG2 is not the fastest renderer...

Title: Re: Detail in Deep Shadows (again)
Post by: TheBadger on April 01, 2012, 03:35:43 PM
Yeah, don't give up man.

I know that faking things and trying to get away with things are not ideas I had in my head when I bought TG2. But it is a skill that you can use just about every place in 3D, from what I can tell. And no one can take points away from you for cheating. In fact I think the more you can cheat the smarter you are in this field! Wouldn't you guys agree?

I hate having to find work arounds, but we have to sometimes. Keep going, it look like a cool project.

@Kadri,
You implied that you are able to use other renders with TG2. If so, how? I like the look of TG's renderer, I think it one of the best in terms of quality (from what I can see). But if I have other options I would like to know about them.

Title: Re: Detail in Deep Shadows (again)
Post by: Kadri on April 01, 2012, 03:59:56 PM
Quote from: TheBadger on April 01, 2012, 03:35:43 PM
...
@Kadri,
You implied that you are able to use other renders with TG2. If so, how? I like the look of TG's renderer, I think it one of the best in terms of quality (from what I can see). But if I have other options I would like to know about them.

I think Tg2 is the renderer i like the most .
There are so many more renderer's out there that i will not and can not try all of them and not mentioning that i could buy them too...

Anyway  :)

Tg2 is very good at outdoor scenes . I tried to make one in Lightwave and it was not easy to get the same lighting-feeling.
In TG2 you put a landscape(dem,ter etc) and 1 or 2 objects and voila you can get a decent scene very fast and easy from an lighting stand point at least.
Artistic quality is another beast of course as all we here know.
Maybe this property  is what makes the outside view of some to think that TG2 images are not artistic ones (cg talks dealing ...) .

In the end you have to choose what suits your aproach the best TheBadger.

I will use TG2 and Lightwave together where i can . I like both.
You can put something like AfterFx to this and this  would be all to make most of what you want to do.
But if you are in a deathline and have to make it very fast you have to choose things you normally maybe would not do.
There are guys who would make some matte paintings , put them on Bilboards and would make the scene Richard wants to make more from 2D then 3D with very little 3D parts.
There is no one way or one software approach to say it is right. It is up to you and your workflow.

Title: Re: Detail in Deep Shadows (again)
Post by: TheBadger on April 01, 2012, 04:18:41 PM
Thanks Kadri, I agree with everything you said. But I think I should ask my question a different way...
Can I render my TG work in a third party renderer? Not part of it, but all of it as if it were in TG. For example, can I through some king of plugin, use lux or mentalray as if they were the renderer that came with TG2?
Title: Re: Detail in Deep Shadows (again)
Post by: Kadri on April 01, 2012, 04:30:13 PM

There is a plugin for max somewhere here but i do not know how much it automatic does .
So far i know full scene export and rendering is not possible.
You have to do it manually with some fiddling .
And getting the lightning done like in TG2 is a manual task because all renderers are different.
So to put it is not one button push. You have to know the other app. very good to make the same lighting i guess .
It depends on your scene etc. but i would make some things definitely in  TG2  then in others and comp only over it .
Especially if i can use a renderfarm :)


Title: Re: Detail in Deep Shadows (again)
Post by: cyphyr on April 01, 2012, 04:43:47 PM
QuoteFor animation you have to cut corners. I think you have still the "perfect still image" mentality Richard  Wink

Yes I think this is very true. Try not to fall in to the trap but still do. The billboard trick "might" work, its worth a try but I suspect that it would produce the results I'm trying to get away from, ie, the flat bill board look.

I think the main issue here is workflow. I was trying to overcome the rendering issues I knew I was going to face before they actually arose. The deep shadows and long volumetric rays were certain to cause problems as was the extremely high population count. I wanted to know it was actually feasible before I properly started the project. However since all the issues are so view dependent I may not have actually learnt that much that's particularly relevant in this process.

For example if I simply accept that areas in deep shadow will have very much less detail in them I could get away with using billboards in those places. This could potentially save a huge amount of render overhead. Also once the city elements are incorporated the populations will reduce considerably. I don't think using only one model is viable in this case, a lot of its going to be fairly close to the camera.

Doing this as a camera projection/2.5d/matte painting ... Well that would be a huge learning curve. Possible I'm sure, and if I was ILM or Weta I'm sure the way I'd go. (I'd have a decent comp artist on hand for a start ;) ) Its a subject I'd really like to learn about though.

QuoteIn fact I think the more you can cheat the smarter you are in this field! Wouldn't you guys agree?
Absolutely.

Ok, not giving up yet then :) It will be a slow project though, lot on at the moment.

Cheers

Richard
Title: Re: Detail in Deep Shadows (again)
Post by: Kadri on April 01, 2012, 04:53:16 PM
Quote from: cyphyr on April 01, 2012, 04:43:47 PM

Ok, not giving up yet then :)
...

Way to go Richard  :D

Curious when we can see this already !
Title: Re: Detail in Deep Shadows (again)
Post by: TheBadger on April 01, 2012, 05:12:45 PM
Quote from: Kadri on April 01, 2012, 04:30:13 PM
...So far i know full scene export and rendering is not possible.
You have to do it manually with some fiddling
And getting the lightning done like in TG2 is a manual task because all renderers are different....it is not one button push. You have to know the other app...

This makes me sad  :-[ I would like the extra freedom, preferably in a nice neat, easy to use package. Lux render is a stand alone, I thought that mental ray was to. I wish some programers out there would develop these things. It could only help to grow the user base for TG2.

Quote from: Kadri on April 01, 2012, 04:53:16 PM
Quote from: cyphyr on April 01, 2012, 04:43:47 PM

Ok, not giving up yet then :)
...

Way to go Richard  :D

Curious when we can see this already !

+1
Title: Re: Detail in Deep Shadows (again)
Post by: TheBadger on April 01, 2012, 05:31:07 PM
Quote from: cyphyr on April 01, 2012, 04:43:47 PM
...Doing this as a camera projection/2.5d/matte painting ... Well that would be a huge learning curve. Possible I'm sure, and if I was ILM or Weta I'm sure the way I'd go. (I'd have a decent comp artist on hand for a start ;) ) Its a subject I'd really like to learn about though...

Hi again cyphyr

If you use after effect or have access, I highly recommend "The Digital Matte Painting Handbook" by David B. Mattingly : http://www.amazon.com/The-Digital-Matte-Painting-Handbook/dp/0470922427
It covers 2.5d in depth. And comes with several comprehensive projects (all files and parts). A guy like you could get through it all in a week or two. And I'm sure you would be able to do the things you want as a result. + with http://www.videocopilot.net/tutorials/ your all set!
I would love to see the things you did as a result!

just sharing, sorry for hijacking  ;D
Title: Re: Detail in Deep Shadows (again)
Post by: Oshyan on April 01, 2012, 11:58:57 PM
The shortest and clearest answer is no, you cannot use another renderer to render TG2 scenes. The 3DS Max plugin only imports/exports a limited set of data (terrain, basic sun position, camera animation). It misses the majority of what makes TG unique, namely procedural texturing, atmosphere model, and volumetric clouds. Also keep in mind that even if you could use a 3rd party renderer, there's nothing to say it would be faster or better than TG *at doing these kinds of scenes*. There's a reason there are dedicated products like TG, if the other products could do the same level of detail and quality with a lower render time, TG wouldn't really need to exist. ;)

- Oshyan
Title: Re: Detail in Deep Shadows (again)
Post by: TheBadger on April 04, 2012, 02:00:54 AM
Quote from: Oshyan on April 01, 2012, 11:58:57 PM
The shortest and clearest answer is no, you cannot use another renderer to render TG2 scenes. The 3DS Max plugin only imports/exports a limited set of data (terrain, basic sun position, camera animation). It misses the majority of what makes TG unique, namely procedural texturing, atmosphere model, and volumetric clouds. Also keep in mind that even if you could use a 3rd party renderer, there's nothing to say it would be faster or better than TG *at doing these kinds of scenes*. There's a reason there are dedicated products like TG, if the other products could do the same level of detail and quality with a lower render time, TG wouldn't really need to exist. ;)

- Oshyan

Ok, that sounds reasonable. Mostly I just want everything I do in TG to easily transfer to any other program I use or will use for any reason I may ever have.  ;)
Title: Re: Detail in Deep Shadows (again)
Post by: JimB on April 09, 2012, 04:56:52 AM
Richard, if you're looking for more detail in your veg, add some reflectivity to your foliage just like in the real world. It can make a big difference.
Title: Re: Detail in Deep Shadows (again)
Post by: cyphyr on April 09, 2012, 05:37:46 AM
Hi Jim,
Yes reflectivity is a key to getting good detail. Do you mean specular reflectivity or , err real reflectivity (with or without rtr) or both?
I  try to start my vegetation scenes with translucency set to 0.5 and the spec set to about 0.2 and tweak from there.
Cheers
Richard
Title: Re: Detail in Deep Shadows (again)
Post by: JimB on April 10, 2012, 02:59:27 AM
Richard, I'd experiment by making all of your foliage diffuse black, and a black layer with zero displacement on top of all surface shaders. That could speed up your render, and you can comp it on top if you're not averse to post work. As the foliage probably already has specularity, I'd say don't add any more. You might not even need GI switched on for a reflection pass if it's just about adding a little extra into the darks. That said, it might add nothing at all  ;)