Planetside Software Forums

Support => Terragen Support => Topic started by: seanQuixote on March 30, 2010, 06:05:43 PM

Title: Terrain and shadows changing during animation
Post by: seanQuixote on March 30, 2010, 06:05:43 PM
I'm wondering if anyone has come across this issue and found a way to defeat it:

New to terragen so I made a simple mountain landscape, added a slight off white shader for snow, put in some key frames to do a fly through and rendered it out.  First try and I got some shadows flickering.  I managed to fix that by switching to from GI to AO.  What I haven't been able to overcome are these slight changes in surface detail from frame to frame. A small shadow will disappear between two frames or a small speck of snow on a small ledge will disappear for a few frames then come back a few frames later.  The over all effect is these intermittent spots in various places throughout the scene that turn on and off as the animation progresses which is pretty distracting.

I've done over 32 test renders with various combinations of settings but I can't seem to get it right.

If anyone else has come across this I'd really, really... really appreciate some insight on how to fix it.

I'm attaching the .tgd file... in case some generous soul wants to take a look.

Title: Re: Terrain and shadows changing during animation
Post by: Tangled-Universe on March 31, 2010, 02:23:06 AM
The detail blending is set to 0.
For still images this is just fine, but for animations it is key to increase it up to 1 max.
Decrease the blur radius of your GI and start testing again. Perhaps you don't need it anymore @ 100, so the GI will look better, which made me notice that you've set GI relative detail to 3 and sampling to 2. That's pretty high if you'd ask me.
If you need detailed GI then I'd suggest to render with GI relative detail at 2 and sampling at 4.
This gives pretty much the exact same result, but is much faster than GI 3/2.

Martin
Title: Re: Terrain and shadows changing during animation
Post by: seanQuixote on March 31, 2010, 10:35:07 AM
Thank you so much... I'll try that.
Title: Re: Terrain and shadows changing during animation
Post by: Oshyan on April 01, 2010, 02:14:45 AM
Agreed, Detail Blending should be at 1 for animations. You may also need to use Ray Detail Region Padding in the Render Settings Advanced tab, if you seem to be getting intermittent shadows from terrain that is out of frame. If so try Detail In Camera with a value of 1.

- Oshyan
Title: Re: Terrain and shadows changing during animation
Post by: MKE on February 28, 2011, 03:28:24 PM
Hello Everyone,

this is my first post here, so I hope I don't talk too much nonsense...

My big boss bought two very nice cinema beamers for 3D projection but we had nothing really nice to show in 3D up to now so this is my job now. I render an animation of a circular flight above a digital elevation model created by a colleague and I think the single images already look quite nice. Just when I started with the animation, I ran into the flickering shadows problem as several people before me (Sunlight a 10° elevation).

First of all: Thanks to all of you in the forum. With your help I managed to reduce errors step by step. On the other hand that means that I was quite astonished and not very happy about the fact that there doesn't exist a manual where you can get all the information from. The node reference is quite a joke at the moment but I hope that it grows step by step. The guys from planetside answer very competent to a lot of questions in the forum. If you would write down these answers to one document and not to a thousend threads you would have a perfect documentation of the software.

The reason why I write:

The shadow problem got much better with "Detail in camera" and a "Ray detail region padding" of 1 (0.1 had no effect, 0.5 was a bit better), but still I had problems that the shadows got smaller when I came nearer. So after trying a lot of other sliders and buttons I tried to enter a number which is outside the "reasonable" values of 0 to 1. With a value of 2 I already was quite happy and with 3 all the shadows stay where they should be. At least I hope, because I restart the complete animation again this evening. My fear about an extremely increase in processing time by inserting big numbers was unnecessary. The needed time is always more or less the same. The trial with a value of 10 was faster than with 3!

So my question: Is there any reason which should keep me from using a very large ray detail region padding (3,5,10,...?) all the time?

Kind regards,
Martin


Quote from: Oshyan on April 01, 2010, 02:14:45 AM
Agreed, Detail Blending should be at 1 for animations. You may also need to use Ray Detail Region Padding in the Render Settings Advanced tab, if you seem to be getting intermittent shadows from terrain that is out of frame. If so try Detail In Camera with a value of 1.

- Oshyan
Title: Re: Terrain and shadows changing during animation
Post by: Oshyan on March 05, 2011, 03:41:14 AM
I would be surprised if a value of 3 or more was not impacting render time significantly. That being said I *think* there may be some limitations which effectively mean anything above a certain value (perhaps 5, not sure) isn't going to make much difference anyway. There have been some recent fixes in internal versions which will be available in a future update to partly address this. But for now you're free to use whatever values you want, as long as you don't see an undesirable render time impact.

As for documentation, yes it's something we're working on, but certainly not quite so simple as collating all our forum answers unfortunately.

- Oshyan
Title: Re: Terrain and shadows changing during animation
Post by: MKE on March 10, 2011, 12:29:17 PM
Quote from: Oshyan on March 05, 2011, 03:41:14 AM
I would be surprised if a value of 3 or more was not impacting render time significantly.

Hi Oshyan,

Thanks for your answer. The new rendering of the animation is running now since about one week with a value of 10 for ray detail region padding (without impact to render time) and with detail=2 (necessary but with factor 3 in rendering time)  and "most" shadows stay where they should be now. Unfortunately "most" means "not all"... I don't know were I should screw to get this fixed. Setting the sun to a higher position in the sky could solve the problem but that would be an ugly solution. The light is very nice as it is now and I like the long shadows as long as they keep still...

I attached some images and my tgd-file. If you have an idea how I could solve the problem I'd be really happy.

Kind regards,
Martin


Title: Re: Terrain and shadows changing during animation
Post by: Oshyan on March 11, 2011, 01:36:01 AM
I'm afraid I won't be able to reproduce the problem without your terrain file. If possible can you email it to support AT planetside.co.uk?

Thanks,

Oshyan
Title: Re: Terrain and shadows changing during animation
Post by: Matt on March 12, 2011, 04:46:10 AM
There is a problem with shadows cast from behind the camera plane. No amount of ray detail region padding can solve this completely, I'm sorry to say. I am considering some possible ways to fix this.

Matt
Title: Re: Terrain and shadows changing during animation
Post by: goldfarb on March 14, 2011, 04:55:03 AM
I've downloaded this file and rendered it with the suggestions given in this thread and still get some flickering...

would it be possible for planetside to edit the settings as best they can to reduce this and then provide it as a guide for these kinds of renders - I realize that different scenes will require different settings but many of us would find an example that works as well as possible very handy to use as reference.
Title: Re: Terrain and shadows changing during animation
Post by: Oshyan on March 14, 2011, 11:46:25 PM
Quite honestly animations demand a lot more tweaking and in many cases specific settings than stills. In this case I actually finding that "Raytrace Everything" is giving the best results (yes, even with a scene composed entirely of terrain), however this is not something we would normally recommend! So the recommendations would really not be good to extrapolate to other scenes.

If my Raytrace Everything experiments work out I'll post the TGD. I did change some other settings, and interestingly it also reduced render time to less than half of what it was.

- Oshyan
Title: Re: Terrain and shadows changing during animation
Post by: Oshyan on March 15, 2011, 12:17:03 AM
Update: Raytrace Everything also did not resolve the problem. For now this seems to be something that can't be fully addressed but we're working on fixes for a future update.

- Oshyan
Title: Re: Terrain and shadows changing during animation
Post by: MKE on March 15, 2011, 05:38:51 AM
Quote from: Oshyan on March 15, 2011, 12:17:03 AM
Update: Raytrace Everything also did not resolve the problem. For now this seems to be something that can't be fully addressed but we're working on fixes for a future update.

Hi Oshyan,

thanks for the troubleshooting. Too bad that there is no fast solution. So I hope that it can be solved for an update. I will not render the whole animation again with a higher sun. The computer is running now for two full weeks and I don't want to throw that all away. I'll finish this animation as it is and hope for the update to render the problematic images again. For the next animations I will use a higher sun and hope for the best.

Kind regards and many thanks for the fast support,
Martin
Title: Re: Terrain and shadows changing during animation
Post by: Oshyan on March 15, 2011, 10:14:40 PM
Here's the TGD for the Raytrace Everything version. Not bad results really, and much faster render time.

By the way you might want to consider turning on the Fractal Detail. I understand accuracy may be an issue in this animation, but I suspect the *lack* of fractal detail is actually contributing to the shadow issues and possibly even the terrain instability with shadows from behind the camera. Something to look at...

- Oshyan
Title: Re: Terrain and shadows changing during animation
Post by: MKE on March 16, 2011, 03:51:12 PM
Hi Oshyan,

thanks for the modified tgd-file. As far as I see you changed:
detail = 0.5 instead 2
ray trace atmosphere = 1 instead 0
GI relative detail = 2 instead 1
GI sample quality = 3 instead 2
GI blus radius = 20 instead 50
detail_jittering = 0 instead 1
ray trace everything = 1 instead 0
minimum threads = 8 instead 1
maximum threads = 8 instead 16
size of subdiv cache in MB= 800 instead 400
ray detail region padding = 1 instead 10

Is there anything I missed? What about the number of threads: Isn't that a problem if I only have two processors?

Unfortunately the quality with detail=0.5 is not really what I want to have. See attached an image example. I also tried with detail=1 and it's better (but not as good as the previous images) but then there's not much time saved any more. Hmmm, I'll do some more tests and then decide how to proceed.

Fractal detail will still be switched off. I want to show the real details of the new DEM and not create more "details" for nicer images. So I can just hope that I don't run into problems because of this. But for the work I do, artificial "details" is a no-go.

Thanks for your help, kind regards,
Martin


Quote from: Oshyan on March 15, 2011, 10:14:40 PM
By the way you might want to consider turning on the Fractal Detail. I understand accuracy may be an issue in this animation, but I suspect the *lack* of fractal detail is actually contributing to the shadow issues and possibly even the terrain instability with shadows from behind the camera. Something to look at...

- Oshyan
Title: Re: Terrain and shadows changing during animation
Post by: Oshyan on March 17, 2011, 01:33:56 AM
Strange, I am not getting that same result. What frame is that? Is it the same settings I posted in the TGD above, or changed?

- Oshyan
Title: Re: Terrain and shadows changing during animation
Post by: MKE on March 17, 2011, 05:00:39 AM
Quote from: Oshyan on March 17, 2011, 01:33:56 AM
Strange, I am not getting that same result. What frame is that? Is it the same settings I posted in the TGD above, or changed?

- Oshyan

Hmmm, I didn't change the settings. The only thing that is different is that I used the camera for the left eye (3D rendering). One thing that could create a difference: You used version 2.3.15.0 for your tests, while I still run 2.2.23.1 here. Maybe I should get the newest version first?

Edit: It seems I have the newest version available... Could you check if the results are the same if you use 2.2.23.1 instead of 2.3.15.0? If this makes the difference I'd be very happy if you could release 2.3.15.0 asap.

Kind regards,
Martin

Title: Re: Terrain and shadows changing during animation
Post by: MKE on March 23, 2011, 05:13:31 AM
Hi Oshyan,

I did some more tests during the last days. I raised the sun to see if the shadow problem disappears. Unfortunately it does not. I went up to 25° over the horizon and the shadows still retreat if you approach. Of course the effect is less because the shadows are not that long any more but it's still visible and annoying. I think that this is a bug in the software which should be fixed as soon as possible. Since the raised sun didn't solve the problem but destroyed the beauty of the DEM with long shadows I set the sun back to 10° above the horizon as it was before.

I wonder why I don't have any problems any more with the little shadows in bright sunlight that should not exist and disappeared by approaching in my first animation trial. This problem is not existent any more. Did you change some hidden values in the tgd-file you uploaded in the forum? If that's the case: Good work. These settings should be also used for the initial settings in a new tgd-file.

In some other tests I saw that the higher values for global illumination slow down the rendering very much without increasing the quality adequately. Instead I increased the picture size to the full resolution (2048 x 1080) by reduced rendering quality settings. This still looks better and is faster than my old and your proposed settings. Maybe it will help other users later on with their animations, so here are my actual settings:

Quality:
Detail = 1 (absolute minimum for decent images)
Anti-Aliasing = 2
Ray trace objects
Ray trace atmosphere
GI relative detail = 1
GI sample quality = 1
GI blur radius = 20
Supersample prepass

Extra:
Pixel filter: Narrow cubic
Anti-aliasing bloom
Detail blending = 1
Displacement filter = 1
Microvertex jittering
(Detail jittering off)
(Lock subdiv to frame off)
Do ray traced shadows
Ray trace everything (which made the rendering much faster)

Advanced:
Minimum threads = 2
Maximum threads = 8
Size of subdiv cache in MB = 800
Ray detail region: Detail in camera
Ray detail region padding = 3
GI prepass padding = 0


Kind regards,
Martin
Title: Re: Terrain and shadows changing during animation
Post by: Matt on March 26, 2011, 02:14:39 AM
Oshyan probably created a "render subdiv settings" node in the render node's internal network. With that node he probably changed a couple of switches and increased the "ray detail multiplier". You would need to keep that node in order to match Oshyan's results.

Unfortunately, I don't think 2.2.23.1 supports all of those settings. You might have seen some warnings when you loaded or copied his render node. Without the "ray detail multiplier" your renders would have lost a lot of detail when you enabled "ray trace everything", which is why you needed to put the master detail up to 1.
Title: Re: Terrain and shadows changing during animation
Post by: Matt on March 26, 2011, 02:17:14 AM
There is nothing you can do to completely prevent popping shadows if the shadows are coming from behind the camera (at more than 90 degrees from the camera viewing direction in 3D space). This is something we need to fix on our end.
Title: Re: Terrain and shadows changing during animation
Post by: MKE on March 28, 2011, 08:24:09 AM
Quote from: Matt on March 26, 2011, 02:14:39 AM
Oshyan probably created a "render subdiv settings" node in the render node's internal network. With that node he probably changed a couple of switches and increased the "ray detail multiplier". You would need to keep that node in order to match Oshyan's results.

Unfortunately, I don't think 2.2.23.1 supports all of those settings. You might have seen some warnings when you loaded or copied his render node. Without the "ray detail multiplier" your renders would have lost a lot of detail when you enabled "ray trace everything", which is why you needed to put the master detail up to 1.

Hi Matt,

yes, there were some warnings. So I hope on the new version of TG2 for the next trial. I will download the new version when the actual animation rendering is finished. It's kind of an iterative process to improve the animation and each time I learn something new or the software has improved... ;-)

Kind regards,
Martin
Title: Re: Terrain and shadows changing during animation
Post by: PabloMack on April 24, 2011, 05:03:59 PM
One thing that I have realized is that, as fractal geometry detail is determined by its distance from the camera, geometry flicker is usually not bad when, say, a mountain peak is far away from the camera. However, the sun can cast a shadow of a distant peak onto some terrain that is very close to the camera. In such cases, the geometry flicker's shadow can be very close to the camera. Magnified this way it can look very bad.

I suspect that all terrain and most shadow flicker is due to TG2 creating geometries from scratch on every frame based on where the camera is during that frame. This obviously stems from TG2's roots in still imagery as animation was an add-on at some later date.

If TG2 had a little box that you could check that forced all fractal geometry to be created up front based on its closest distance to the camera's path (i.e. always used the distance to the camera at its nearest point from any one point in the geometry) then all geometry flicker and most shadow flicker (that due to geometry and not due to variation in light sampling) could be eliminated. In fact, by generating geometry only once (instead of for every frame), the whole process could, perhaps, be sped up. Granted, there would be excess geometry not needed by every frame. But the alternative is to have to generate excess geometry for the whole frame just to eliminate bits of flicker here and there as it is now.
Title: Re: Terrain and shadows changing during animation
Post by: Matt on April 24, 2011, 07:19:37 PM
Quote from: PabloMack on April 24, 2011, 05:03:59 PM
One thing that I have realized is that, as fractal geometry detail is determined by its distance from the camera, geometry flicker is usually not bad when, say, a mountain peak is far away from the camera. However, the sun can cast a shadow of a distant peak onto some terrain that is very close to the camera. In such cases, the geometry flicker's shadow can be very close to the camera. Magnified this way it can look very bad.

I suspect that all terrain and most shadow flicker is due to TG2 creating geometries from scratch on every frame based on where the camera is during that frame.

Yes that's true.

Quote
If TG2 had a little box that you could check that forced all fractal geometry to be created up front based on its closest distance to the camera's path (i.e. always used the distance to the camera at its nearest point from any one point in the geometry) then all geometry flicker and most shadow flicker (that due to geometry and not due to variation in light sampling) could be eliminated. In fact, by generating geometry only once (instead of for every frame), the whole process could, perhaps, be sped up. Granted, there would be excess geometry not needed by every frame. But the alternative is to have to generate excess geometry for the whole frame just to eliminate bits of flicker here and there as it is now.

The needed geometry would be huge. In many cases I expect it would be unmanageable.

I have a solution to the shadow flickering which we will release with Terragen 2.4 with Animation. Actually it's a combination of various recent improvements which haven't been made into official settings yet, combined with a new ray detail stabilisation feature which is working very well. This solves problems with small micropolygon shadows changing from frame to frame. There is still the second problem of shadows cast from terrain behind outside the camera view (which ray detail padding helps with but doesn't work for terrain behind the camera). We'll also provide a solution to that in v2.4.

Matt
Title: Re: Terrain and shadows changing during animation
Post by: MKE on July 28, 2011, 09:28:36 AM
Quote from: Matt on April 24, 2011, 07:19:37 PM
I have a solution to the shadow flickering which we will release with Terragen 2.4 with Animation. .... There is still the second problem of shadows cast from terrain behind outside the camera view .... We'll also provide a solution to that in v2.4.


Hi Matt,

this is very good to hear and absolutely neccessary to get high quality animations. During the last weeks I played around with differend parameter values (and I got a faster computer... ;-) so the processing time for good quality animation frame is down to 8 minutes now.

There are still some errors in the animation. Two error sources are hopefully gone with TG 2.4, for a solution of the third I hope on the expertise of you and the forum.

- there are still receding shadows as you approach them. Other than in my previous animation this time the sun is at a 90° to the camera look angle (and the flight path). The camera is looking towards north and the sun is 10° over the horizon directly in the east. See attached subsets of image 855 and 856.

- The shadow flickering is quite annoying, although I increased the GI values. I really hope for TG 2.4.
See attached subsets of images 1142 and 1143.

- From time to time it looks like the indirect illumination of the atmosphere is switched of. Then after some images, the normal illumination comes back. Do you know about this bug? Do you have any idea how to deal with that? See attached subsets of the images 1537 and 1538.

The parameter values I use now (tgd-file attached):

Quality:
Detail = 1
Anti-Aliasing = 4
Ray trace objects
Ray trace atmosphere
GI relative detail = 2
GI sample quality = 2
GI blur radius = 200
Supersample prepass

Extra:
Pixel filter: Narrow cubic
Anti-aliasing bloom
Detail blending = 3
Displacement filter = 0
Microvertex jittering
(Detail jittering off)
(Lock subdiv to frame off)
Do ray traced shadows
Ray trace everything

Advanced:
Ray detail region: Detail in camera
Ray detail region padding = 3
GI prepass padding = 0



By the way: the forum editor is horrible. I had to type blind because while typing the visible window sticks to the uppermost text. And when sending the post there came the error message that tif-images are not allowed and all my text was lost. So now I type everything again in my own editor and copy it into the forum...

Kind regards,
Martin[/quote]
Title: Re: Terrain and shadows changing during animation
Post by: dandelO on July 28, 2011, 01:14:35 PM
I don't know if anyone's posted this to this thread yet, just read the last reply.

To get better quality edges and shadows of planet geometry, you can(if you have the render subdivision settings node enabled in the renderer), check the box 'fully adaptive'. The difference between the following two quick renders is only that single setting...

Detail=0.75(detail blending=0)
Fully adaptive=unchecked(default):
[attachimg=#]

Fully adaptive=checked:
[attachimg=#]

Checking fully adaptive makes(according only to my personal tests, I've no actual confirmation of that or that this is what the setting does) cast shadows and displacement edges render more correctly. I've tried this with animated parameters too. With 'FA' checked, the results of shadows are a whole lot better(at the expense of some render time!).

When the 'AA adaptive sampling' methods were introduced, I noticed the shadows became more raggedy at default settings, I know there was always a shadow popping problem before this but it was definitely exacerbated, I think, with the introduction of some of these settings a few releases ago.

The 'ray detail multiplier' is set at default=0.25 for both of the above, btw.
Title: Re: Terrain and shadows changing during animation
Post by: Henry Blewer on July 29, 2011, 04:16:51 AM
I can't find this shadow setting you are talking about. Where is it?
Title: Re: Terrain and shadows changing during animation
Post by: MKE on July 29, 2011, 05:07:05 AM
Quote from: dandelO on July 28, 2011, 01:14:35 PM
... if you have the render subdivision settings node enabled in the renderer...

The improvement in your is really impressive. Thanks for showing us the difference.

Well, I have this subdiv setting node for one renderer (@njeneb: Node Network tab, then on the left side you see a list of nodes where the renderers have a "+" in front of them. If you click on that, some additional settings become visible). I have no idea why only one renderer has this node and how I could manage to have it also at the renderer I use for the animations. In the node palette (View->Node Palette) I find no render nodes to chose from. Right mouse click? Nothing... No possibility to "copy" or "create" a node. Am I to stupid to use TG2?

How can I copy or create a new render subdivision node for the renderer I actually use?

Edit: Here two new images showing the indirect light problem. Nothing has been changed in the tgd-file except the camera position...

Kind regards,
Martin
Title: Re: Terrain and shadows changing during animation
Post by: Henry Blewer on July 29, 2011, 06:27:54 AM
I do not seem to have this.
Title: Re: Terrain and shadows changing during animation
Post by: dandelO on July 29, 2011, 07:21:39 AM
Henry; To enable these settings visit the internal network of your render node and just copy/paste(ctrl+c/ctrl+v) the following clipfile text...

<terragen_clip>
  <render_subdiv_settings>
  </render_subdiv_settings>
</terragen_clip>

Now you have access to that setting, among others. You might want to save your default project with this node enabled so you don't need to repeat this process each time you need it. :)
Title: Re: Terrain and shadows changing during animation
Post by: Henry Blewer on July 29, 2011, 10:45:30 AM
Thanks Martin. I wonder how many other things are hidden?
Title: Re: Terrain and shadows changing during animation
Post by: MKE on July 29, 2011, 11:47:35 AM
Quote from: dandelO on July 29, 2011, 07:21:39 AM
<terragen_clip>
  <render_subdiv_settings>
  </render_subdiv_settings>
</terragen_clip>

That's it! In my case it's not "fully adaptive" which makes the difference but "Force all edges". This one should really always switched on when rendering animations!!! "Jitter shading points" had no effect at all in my test images and "Fully adaptive" resulted in a little bit more speckle than without it so I left it away.

I now started testing to reduce the detail values and it seems that I can get away with detail=0.8 instead of 1.0 . That's 5 minutes and 30 seconds per frame instead of 8 minutes. That's a huge difference together with an improved image quality. Great.

And look at the images. It seems as if the problem of the on-and off switched indirect light has gone too. Greatgreatgreat...

Kind regards,
a happy Martin

... who now has one week off and has to be patient to see the new animation when he comes back...
Title: Re: Terrain and shadows changing during animation
Post by: Matt on August 02, 2011, 12:17:39 AM
Hi dandelO,

Fully adaptive ON is the default setting. The hidden node allows you to turn it off (which can help with animations), but as you've shown it is not always a good idea to do so.

Matt
Title: Re: Terrain and shadows changing during animation
Post by: Tangled-Universe on August 02, 2011, 05:37:48 AM
Hi Matt,

Could you elaborate a bit why these particular settings can help with animations?
Does "fully adaptive" has to do with reducing subdivisions where not many are needed?
What does "force all edges"?
I suppose the jittering of shading points can result in shading differences between frames and thus some flickering/popping.
So far in my experience switching this off is really useful.

Cheers,
Martin
Title: Re: Terrain and shadows changing during animation
Post by: Henry Blewer on August 02, 2011, 08:08:11 AM
I noticed the odd shadows on a test anim. The one I did which lasted 30 seconds I worked around this by deleting the enviro light and using a couple fill sun lights.
Title: Re: Terrain and shadows changing during animation
Post by: dandelO on August 02, 2011, 01:35:59 PM
I guess I must have saved my own file with 'FA' unchecked at some point, or the one that I salvaged from a .tgd in here already had it unchecked when I saved it or something.

I have tried animating a set of frames as comparison between with and without 'fully adaptive' and I didn't ever notice any benefit to unchecking it, other than the faster render times. I wonder, under what circumstances would unchecking it be beneficial? I only ever noticed a nasty edge and ragged shadow without it on before. The scene I used to test was a relatively close up shot of stones with animated warping in the surface shaders(liquid stones, was quite nice, and strange), maybe it works best to uncheck FA when the scene is very large scale and small shadow details aren't as important, instead?

But, when I first noticed that shadows had become more ragged at the default render settings was when I rendered this ant that Marc made some time ago against a white background (http://forums.planetside.co.uk/index.php?topic=9668.msg101484#msg101484). I remember thinking, why are my shadows looking so horrible when I'm using a good detail and AA setting. I knew nothing of the render subdivision settings node at that time, just over a year ago it seems. I think the pixel filters had just been included around that time but there were no editable AA sampling settings... I think.
Title: Re: Terrain and shadows changing during animation
Post by: MKE on August 08, 2011, 05:15:59 AM
Quote from: MKE on July 29, 2011, 11:47:35 AM
I now started testing to reduce the detail values and it seems that I can get away with detail=0.8 instead of 1.0 .

The testing went on and on at that evening and I ended with quite different values as before. Mostly because "GI sample quality" changes did not lead to a steady improvement at lower values. The shadows over snow were flipping from bright to dark when I changed the GI sample quality by one and only stabilised when using real high values. On the other hand "GI relative detail" didn't have much effect but slowing down the processing considerably.

I ended up with the following values:

Quality:
Detail = 0.8
Anti-Aliasing = 4
Ray trace objects
Ray trace atmosphere
GI relative detail = 1
GI sample quality = 10
GI blur radius = 200
Supersample prepass

Extra:
Pixel filter: Narrow cubic
Anti-aliasing bloom
Detail blending = 3
Displacement filter = 0
Microvertex jittering
(Detail jittering off)
(Lock subdiv to frame off)
Do ray traced shadows
Ray trace everything

Advanced:
Ray detail region: Detail in camera
Ray detail region padding = 3
GI prepass padding = 0

Render subdiv settings:
(Full adaptive off)
Roce all edges
(Jitter shading points off)
Ray detail multiplier: 0.25


The animation looks very fine with a lot of details, smooth edges and no pixel flipping in the background any more. The shadows keep stable from image to image. Just one thing didn't improve: I still get some on and off switching of the indirect light when the camera is high above the ground. See attached three images where the first is ok, the second has less indirect light and the third none at all any more. This must be a software bug because the values during the animation didn't change at all. Matt, Oshyan, do you know how to handle this?

Kind regards,
Martin
Title: Re: Terrain and shadows changing during animation
Post by: Tangled-Universe on August 08, 2011, 05:53:15 AM
I'm wondering why Oshyan enabled "raytrace everything" for you.
It explains directly why your terrain looks inferior to your previous tests, because the ray detail multiplier is set at 0.25.
That means that if you render at 0.8 detail with the raytraced renderer, then the effective render detail will be 0.8 x 0.25 = 0.2!
Consequently that might be the reason why you also need that insane value for detail blending @ 3, which also slows things down.

This is what I'd do for render settings:

Detail 0.7
AA4
Raytrace atmosphere/objects enabled
GI relative detail @ 1
GI sample quality @ 6
GI Blur radius @ 40
SS-prepass enabled

Pixel filter: Narrow cubic (shouldn't matter that much I suppose)
Anti-aliasing bloom (shouldn't matter that much I suppose, post-render effect I'd say)
Detail blending = 1
Displacement filter = default (can't remember I ever touched this for animation, so keep it at default, though I can't remember whether default = 0 or 1)
Microvertex jittering enabled
Detail jittering disabled
Lock subdiv to frame disabled
Do ray traced shadows
Ray trace everything disabled

Advanced:
Ray detail region: Detail in camera
Ray detail region padding = 3 (that's high, but considering your problems it might be necessary)
GI prepass padding = 0

Render subdiv settings:
Fully adaptive enabled
Force all edges enabled
Jitter shading points disabled
Ray detail multiplier: 0.25 (this setting won't have an effect anymore on your terrain, since you disabled raytraced terrain rendering in the extra tab)

With these settings I recently renderer a far more detailed and displaced terrain than your project, so these settings should at least get you in the right direction.

The latest alpha contains a new advanced feature which should allow TG to deal better with the shadows from behind camera and stuff.
Don't know when that will be available.

Another question:

Are you testing your animation settings at the resolution you're posting?

Cheers,
Martin

p.s. you can send me your project file and I can help you along with finding settings, if you'd like.
Title: Re: Terrain and shadows changing during animation
Post by: dandelO on August 08, 2011, 08:32:12 AM
I'd like to add/contradict something here.

I've been working on a play-scene(TG generated skatepark) for fun.

I found the settings I needed to use are the exact same settings as Martin describes for the render subdivision settings(without RTE), all save for one. The detail multiplier DOES actually significantly alter the outcome of non-ray-traced terrain.

See these small crops of some sharp areas displaced from the planet surface;

RDM=0.25
[attachimg=#]

RDM=1
[attachimg=#]

The way I thought it would work would be just as you described, Martin but it actually isn't. The RDM does have an effect, even when all logic(and an almost confirmation from Oshyan in my 'rescale noise' thread) says to me that it shouldn't.
I'm kind of losing my mojo with TG at the moment, this scene was just to actually make me use it and get my finger out of my arse but all these questionable settings and inconsistencies, even just the lack of explanation on what long-in-place settings do(intersect underlying for example, anyone? I know how to work it my way but, that's it), are getting to be too much work to be fun.

Again, what would be the point in un-checking 'fully adaptive' in an animation?
Title: Re: Terrain and shadows changing during animation
Post by: Tangled-Universe on August 08, 2011, 08:56:25 AM
Hi Martin... (wow man, there are actually 3 Martin's in here ;D)

That's definitely as well as adding as disturbing/contradicting what you say now.
Need to check that at home. We both have a history of finding odd things which eventually weren't bugs or odd things ;)
So far I have no idea why the RDM would improve the jaggedness of the terrain and shadows. Well, the terrain's shadows are still raytraced, but it clearly seems that the edges are smoother as well, so it also seems to affect geometry. Weird! It's indeed contradictive to what we've always been told.
OTOH, the displacements are so steep/unusual that this might be the reason why the raise in RDM helps, whatever the reason it is. It might be an exceptional situation.


I understand your pain and agony. I thought you are an alpha tester as well?
On the alpha forums there's a 2 part tutorial on Intersect Underlying. The first part covers the basics. The second part is announced/promised.
It took Matt over a year to do it, so it might take another year for him to make the second tutorial ;)

I'm still awaiting his response to my answer my questions related to "fully adaptive" and "force all edges".
Seems I'm not alone anymore.

Earth to Matt ;)

Cheers,
Martin
Title: Re: Terrain and shadows changing during animation
Post by: dandelO on August 08, 2011, 09:50:25 AM
It doesn't seem to just happen with really sharp edges, Martin, I've noticed the difference with the RDM on simple fake stones shader's displacement as well.
Just a clear explanation on the more obscure setting's functions would be adequate, of course, the subdivision settings aren't actually meant to be out there yet but there are many other areas where I find contradictory results and outcomes, for example, what exactly is the relaitionship to the IU functions in line with the smoothing tab(and compute terrain smoothing), so far, it's been officially said that those settings all relate, but not how they do. Similarly, the noise variation methods in probably the most common shader node(power fractal), even in the online documentation, is still missing that information, the node reference for the PFS seem to be the very same ones that Frank wrote around 3-4 years ago and as yet, hasn't been updated.

I think it just comes down to the old story that everyone already knows, there is no solid documentation and labelling of many features. Many of us can work things out by repeated trial and error methods but, not everyone has the time to just sit and render back to back tests checking one setting against another, I think people want to be able to read that this slider does this, that checkbox does that, is all.
Title: Re: Terrain and shadows changing during animation
Post by: MKE on August 08, 2011, 10:37:55 AM
Hi Martin,

thanks for your help. I startet processing frame 30 (the first frame where the indirect light was completely missing) using your values. It's faster than before: 2'28" instead of 4'02". The shadows are ok now so I repeated frame 30 again with my old values, and the shadows become completely black again. Damn... Interestingly the shadows are longer with my old set of values...


I tried again with frame 735 with near foreground and with 1137-1140 with shadows on snow. The details are still ok, just the area in the background changes quite a lot and new kinds of shadows are created. I'll let it run over nigt to see if these shadows change while approaching the terrain. If not, then this is fine. I just saw that your value of 6 for GI sample quality is not enough in my case. The shadows over snow change a bit from frame to frame. I'll stick here to 10. Hmmm, one more thing: After approaching the earth surface, the length of the shadows seem to be more or less the same. That means, the shadows with your values must change when approaching them. I'll see tomorrow.

Concerning resolution, don't be afraid: My renders have 1920x1080 pixels. I just always cut out as little as possible to show the problematic area.

Thanks for offering your help concerning the project file. I'll attach it but let's first look what happens to the animation that will run tonight. I'll write tomorrow about it.

Thanks,
Martin
Title: Re: Terrain and shadows changing during animation
Post by: Tangled-Universe on August 08, 2011, 12:03:05 PM
Quote from: MKE on August 08, 2011, 10:37:55 AM
Hi Martin,

thanks for your help. I startet processing frame 30 (the first frame where the indirect light was completely missing) using your values. It's faster than before: 2'28" instead of 4'02". The shadows are ok now so I repeated frame 30 again with my old values, and the shadows become completely black again. Damn... Interestingly the shadows are longer with my old set of values...


I tried again with frame 735 with near foreground and with 1137-1140 with shadows on snow. The details are still ok, just the area in the background changes quite a lot and new kinds of shadows are created. I'll let it run over nigt to see if these shadows change while approaching the terrain. If not, then this is fine. I just saw that your value of 6 for GI sample quality is not enough in my case. The shadows over snow change a bit from frame to frame. I'll stick here to 10. Hmmm, one more thing: After approaching the earth surface, the length of the shadows seem to be more or less the same. That means, the shadows with your values must change when approaching them. I'll see tomorrow.

Concerning resolution, don't be afraid: My renders have 1920x1080 pixels. I just always cut out as little as possible to show the problematic area.

Thanks for offering your help concerning the project file. I'll attach it but let's first look what happens to the animation that will run tonight. I'll write tomorrow about it.

Thanks,
Martin

Interesting that the shadows become longer. Much longer or just a tad bit longer?

Concerning the changes you describe for frame 735 and 1137-1140.
How does the area in the background change? What do you mean?
Is the rendered outcome different from your previous tests? That would be logical, as you used quite different settings which likely give a different visual result.
Or is the background area changing in terms that the difference between frames is (too) big? Like geometry popping or shadows popping? That would be need to be solved then.

If it is the first issue then don't worry. This is how it looks and that's it. No one will ever know how it otherwise would look, except for you ;)
If you don't like it, then change the design, as this is likely the way you designed it.

You still might try GI 1/8 instead of 1/10. It still could shave off a bit of rendertime.

I'm not sure what you mean with " the shadows with your values must change when approaching them"?

Sorry I had to ask about the resolution ;D In the 4-5 years I'm active here I've seen the weirdest and simplest things done wrong :)
I just had to ask/make sure.

Cheers,
Martin
Title: Re: Terrain and shadows changing during animation
Post by: MKE on August 09, 2011, 03:52:25 AM
Quote from: Tangled-Universe on August 08, 2011, 12:03:05 PM
Interesting that the shadows become longer. Much longer or just a tad bit longer?

Quite a bit. Compare the two images of frame 1, once processed with your parameters and once with mine. I like to use irfanview to flip from one image to the other using the arrow keys, then you can see slightest differences in two images.


Quote
Concerning the changes you describe for frame 735 and 1137-1140.
How does the area in the background change? What do you mean?

Is the rendered outcome different from your previous tests? That would be logical, as you used quite different settings which likely give a different visual result.
Or is the background area changing in terms that the difference between frames is (too) big? Like geometry popping or shadows popping? That would be need to be solved then.

The background is a bit different depending on the way how it is processed. This is why I let the computer run last night to process some 80 frames. Now I'm sure that the differences in the background are no problem because it is consistent from frame to frame, no pixel flipping in the background (as I already had in the past...)

But there is a serious problem: I get again shadows in areas where they should not be and when I approach, they disappear, just look at the subsets of frame 1120-1124.

I think, that's the reason why Oshyan switched on "Ray-trace everything" in the tgd-file some weeks ago... Bad luck, your parameters only need half the processing time.

ReCheers,
Martin
Title: Re: Terrain and shadows changing during animation
Post by: Tangled-Universe on August 09, 2011, 03:58:53 AM
Quote

I think, that's the reason why Oshyan switched on "Ray-trace everything" in the tgd-file some weeks ago... Bad luck, your parameters only need half the processing time.

ReCheers,
Martin

Hmmm...yes that could be the reason, not sure though.

Bad luck? I don't know!
I think this is a good point to start from. Basically the only problem left is the shadow inconsistencies. The length of the shadows, I wouldn't be bother too much about it, but that's me. It works and it's the way it looks. Pragmatic, I know.

So did you render these last 4 frames with my settings or a mix of mine with yours? What's the GI sample quality?

Perhaps you can post the current rendersettings again? I think it's worth to start tweaking from these settings, especially because the rendertimes are significantly shorter.
Title: Re: Terrain and shadows changing during animation
Post by: MKE on August 09, 2011, 04:23:34 AM
Quote from: Tangled-Universe on August 09, 2011, 03:58:53 AM

Bad luck? I don't know!
I think this is a good point to start from. Basically the only problem left is the shadow inconsistencies.

Well, I think you're right. I saw that the glass is half-empty, you see that it's halv-full. That's the better way to look at things... ;-)

Quote
The length of the shadows, I wouldn't be bother too much about it, but that's me. It works and it's the way it looks. Pragmatic, I know.

I'm with you. I don't want to have unreal features in the elevation  model like fractal details but the shadows are just a way to make the DEM look nice. So it doesn't matter if they are a bit longer or not. But they should stay on their place on the different frames. It looks really strange if the shadows crawl away when you come nearer.

Quote
So did you render these last 4 frames with my settings or a mix of mine with yours? What's the GI sample quality?
It's your settings but with a GI sample quality of 10. I did some tests concerning the GI settings some days ago using a smaller render image size. I got the following render times:
GI sample quality  8: 2'39"
GI sample quality  9: 2'40"
GI sample quality 10: 2'43"
GI sample quality 20: 3'07"
This is why I prefer to keep the sample quality at 10. Maybe the whole animation would be good with a value of 8 too, but maybe there is a tricky shadow area again where it makes a diference. And I cannot test all frames before, so I prefer to have the render time increased by 2%...

Quote
Perhaps you can post the current rendersettings again? I think it's worth to start tweaking from these settings, especially because the rendertimes are significantly shorter.
I didn't change the tgd-file in the meantime, it's still the same.
Title: Re: Terrain and shadows changing during animation
Post by: Tangled-Universe on August 09, 2011, 05:29:31 AM
Ok, thanks.

So I'm thinking out loud now: maybe the shadows change in length because the closer by the shadows you come the more screenspace the shadows take.
Consequently more samples would be taken and the shadows accuracy will change over frames.

*If* this is correct we'd need a setting which increases the samples. Before we start raising GI relative detail from 1 to 2 it might be good to test this by actually disabling GI. Does it still occur? If not, then it has to do with GI. If it still is, then not.

I can't stop think about Dandel0's test with the RDM increased to 1 without raytracing the terrain. It did affect shadows and edge accuracy, although we both completely do not understand why.
So after checking the GI, you might give the 4 frames a shot with increased RDM.


Dude, I've never seen someone having so much troubles with his animation :(
But we'll figure it out, eventually :)
Title: Re: Terrain and shadows changing during animation
Post by: MKE on August 09, 2011, 06:07:50 AM
Hi Martin,

without GI the non-existent shadow is still there. So afterwards I run with GI and a RDM of 1.0. And it looks good, but it's rather slow, about 14 minutes.

Quote
Dude, I've never seen someone having so much troubles with his animation
Hmm, I wonder why it should be more troublesome in my animation. It's a normal DEM without any special details so the problems I have should occur everywhere else...

The animation has already improved a lot thanks to the help of different people of the forum but still:


IT WOULD BE MUCH EASIER IF WE HAD A KIND OF HANDBOOK!!!

Title: Re: Terrain and shadows changing during animation
Post by: MKE on August 09, 2011, 09:24:25 AM
I did some tests concerning the ray detail multiplier now. The shadows of my last post already disappear at rdm=0.4 . Unfortunately there are others which are more stubborn. So I need rdm=0.9 to get rid of them all (at least I hope so...). Again, I'll let an animation run over night to see if I'm right.
Title: Re: Terrain and shadows changing during animation
Post by: Tangled-Universe on August 09, 2011, 10:25:32 AM
Quote from: MKE on August 09, 2011, 06:07:50 AM
Hi Martin,

without GI the non-existent shadow is still there. So afterwards I run with GI and a RDM of 1.0. And it looks good, but it's rather slow, about 14 minutes.

Quote
Dude, I've never seen someone having so much troubles with his animation
Hmm, I wonder why it should be more troublesome in my animation. It's a normal DEM without any special details so the problems I have should occur everywhere else...

The animation has already improved a lot thanks to the help of different people of the forum but still:


IT WOULD BE MUCH EASIER IF WE HAD A KIND OF HANDBOOK!!!


The test you did isn't very conclusive.
It is "w/o GI + rdm 0.25" vs "w/ GI + rdm 1", so the beneficial effects cannot be certain because of the GI or RDM alone.
Looking at your previous test it seems to be RDM.

Things start to make sense now, take a look at this quote from the description of RDM at the alpha test forum:
...allows you to control the detail of surfaces underwater, shadow-casting detail etc. See the Alpha Test forum for more information.

So the shadow-casting is being handled by the raytracer anyway it seems. Regardless whether you use ray trace everything (RTE).
With RTE enabled probably only the surface shading will also be rendered with the raytracer.
Increasing RDM makes sense for the shadows now. Why the edges are more smooth in Dandel0's example, don't know yet.
More information didn't show up on the test forum, so this is all.
Title: Re: Terrain and shadows changing during animation
Post by: MKE on August 09, 2011, 11:05:06 AM
Quote from: Tangled-Universe on August 09, 2011, 10:25:32 AM

The test you did isn't very conclusive.
It is "w/o GI + rdm 0.25" vs "w/ GI + rdm 1", so the beneficial effects cannot be certain because of the GI or RDM alone.
Looking at your previous test it seems to be RDM.

Befor this test I had GI and rdm 0.25.
Then the first test was just to switch off GI which had no effect on the shadows.
The second test was then with GI again and a rdm of 1.0 which eliminated the shadows.

So I always changed only one parameter and it's definitely the rdm which helps to avoid the wrong shadows.

Kind regards,
Martin
Title: Re: Terrain and shadows changing during animation
Post by: Tangled-Universe on August 09, 2011, 02:13:40 PM
The naming convention you used for the files clearly said something else, so that's why I made the remark :) peace ;)

How are things going now for the other frames with RDM increased?
Title: Re: Terrain and shadows changing during animation
Post by: Oshyan on August 09, 2011, 06:15:18 PM
When I messed with these files some months back I tried RTE as a sort of baseline comparison (because with a ray detail multiplier of 1 the raytracer is in some ways more "accurate", as I understand it). It turned out to solve some of his problems so I posted the file with that enabled. It clearly didn't turn out to be a comprehensive solution. What has been discovered since about RDM affecting regular shadows makes perfect sense to me, but I didn't think of it at the time. I'm not sure how much more there is to add but I'll have Matt look at the thread again to see if there are any more ideas on his end. As he said before though there are certain limitations which can't be easily resolved at this point and do require core changes.

As to why this animation is seemingly more problematic than others, my guess would that it's actually because it's such a simple scene. In an animation of a more complex scene there is so much for your eye to pay attention to and so much breaking up the shadows, lighting, etc. that it makes relatively small (or even not so small) errors harder to notice. So if your scene had trees and more complex terrain texturing, it would likely make these issues largely unnoticeable. But I understand this look is probably what you need to achieve.

- Oshyan
Title: Re: Terrain and shadows changing during animation
Post by: MKE on August 10, 2011, 05:04:04 AM
Hi Martin and Oshyan,

Oshyan, thanks for telling Matt about this thread. I wonder if "ray trace everything" means the same as "ray detail multiplier = 1.0". When I look at the render time, it could be the case.

Quote from: Tangled-Universe on August 09, 2011, 02:13:40 PM
The naming convention you used for the files clearly said something else, so that's why I made the remark :) peace ;)
??? Yes, peace. But it doesn't say anything else. I just didn't upload the original problematic image with GI on and RDM=0.25, that's all. But that doesn't matter any more, it's clear now that it's not the GI which causes these problems and that with a higher RDM one can solve this problem.

Quote
How are things going now for the other frames with RDM increased?
Just before I wantet to leave office, I noticed that there is still a specular reflection in the shadow which looks unnatural. Therefore I increased the GI sample quality from 10 to 12 and now it's much better and not disturbing any more in the animation. There seems to be no problem any more with retreating and disappearing shadows, that's fine.
On the other hand, the render time gain of your solution melts with every parameter value I increase. It's now 17 minutes per frame compared to 19 minutes as it was before.


This morning I started the processing of the three frames which had the specular reflection inside the shadow again with my former set of values (RTE on), to see, if there is any difference.

Kind regards,
Martin
Title: Re: Terrain and shadows changing during animation
Post by: Tangled-Universe on August 10, 2011, 07:28:02 AM
Quote from: MKE on August 10, 2011, 05:04:04 AM
Hi Martin and Oshyan,

Oshyan, thanks for telling Matt about this thread. I wonder if "ray trace everything" means the same as "ray detail multiplier = 1.0". When I look at the render time, it could be the case.

No as far as I know that does not mean the same.
Remember when your previous settings were detail 0.8 and RDM @ 0.25? That's 0.8 x 0.25 = 0.2 effective detail.
RTE basically tells TG2 to not use the rasterized renderer but the ray traced renderer. Consequently the above equation will be used.
So to have your raytraced terrain detail in line with the detail setting in the renderer you need to set RDM to 1, because then the effective detail rendered will be the same, according to that equation.
I think you can test this yourself with a crop render. Set detail to 1 and RDM to 0.1 and compare that with detail 1 RDM 1. The first should look horrible, the last a lot better.

Quote
??? Yes, peace. But it doesn't say anything else. I just didn't upload the original problematic image with GI on and RDM=0.25, that's all. But that doesn't matter any more, it's clear now that it's not the GI which causes these problems and that with a higher RDM one can solve this problem.

Ah right, now I see, missed that info. Oh well, never mind :)

Quote
Just before I wantet to leave office, I noticed that there is still a specular reflection in the shadow which looks unnatural. Therefore I increased the GI sample quality from 10 to 12 and now it's much better and not disturbing any more in the animation. There seems to be no problem any more with retreating and disappearing shadows, that's fine.
On the other hand, the render time gain of your solution melts with every parameter value I increase. It's now 17 minutes per frame compared to 19 minutes as it was before.

That's a pity :( So basically you only increased RDM to 1.0 and set GISQ to 10-12? That's quite an increase in rendertime unfortunately.

I think Oshyan is right when he said that normally the complexity of a terrain covers things up and that the extreme simplicity in this situation painfully exposes the slightest potential problem which you normally really wouldn't encounter.
I don't know that doesn't help you, sorry.
Title: Re: Terrain and shadows changing during animation
Post by: dandelO on August 10, 2011, 09:38:39 AM
Just for the point of confirmation on my earlier posts in this thread(sorry for the interuption on your animation thread, MKE, but I think it's important to keep this issue here where it was first brought up. :);

I said I'd noted this happening on fake stones as well.
Here's two tests, a simple and entirely default fake stones shader, no surface shaders or other modifications added. Both tests at final render detail=1 for consistency. Both with the default micropolygon renderer.

Firstly, an RDM setting of default=0.25 with fully adaptive enabled;
[attachimg=#]

Lastly, the very same with RDM=1;
[attachimg=#]

The difference is plain to see.

I'm out.
Title: Re: Terrain and shadows changing during animation
Post by: Tangled-Universe on August 10, 2011, 09:40:17 AM
Quote from: dandelO on August 10, 2011, 09:38:39 AM
Just for the point of confirmation on my earlier posts in this thread;

I said I'd noted this happening on fake stones as well.
Here's two tests, a simple and entirely default fake stones shader, no surface shaders or other modifications added. Both tests at final render detail=1 for consistency. Both with the default micropolygon renderer.

Firstly, an RDM setting of default=0.25 with fully adaptive enabled;
[attachimg=#]

Lastly, the very same with RDM=1;
[attachimg=#]

The difference is plain to see.

Scroll back and see why Martin :)
Well, not entirely why, but it definitely explains why the shadows are improved.
The edges is still a mystery so far.

Perhaps you have a lucid idea?
Title: Re: Terrain and shadows changing during animation
Post by: dandelO on August 10, 2011, 09:50:01 AM
Right. I knew rendered shadows were RDM dependant. Not that 'shadow casting detail' was.
This still really confuses me, Martin, as the shadow casting geometry is being rendered with the MP rasterizer and surely every element in a scene that has height and a light behind it will cast a shadow, what if I remove the light or make it directly above, so it won't cast a shadow? ...

So, pretty much every element in a scene is degraded by the default 0.25 RDM setting, all but colour shaders, really. Do I have that right?
Title: Re: Terrain and shadows changing during animation
Post by: dandelO on August 10, 2011, 09:52:25 AM
Or, even uncheck 'do ray traced shadows'? Will this then make the displaced surfaces render correctly?

Do you see what I mean, or am I confusing things further?
Title: Re: Terrain and shadows changing during animation
Post by: Tangled-Universe on August 10, 2011, 10:02:01 AM
Quote from: dandelO on August 10, 2011, 09:50:01 AM
Right. I knew rendered shadows were RDM dependant. Not that 'shadow casting detail' was.
This still really confuses me, Martin, as the shadow casting geometry is being rendered with the MP rasterizer and surely every element in a scene that has height and a light behind it will cast a shadow, what if I remove the light or make it directly above, so it won't cast a shadow? ...

So, pretty much every element in a scene is degraded by the default 0.25 RDM setting, all but colour shaders, really. Do I have that right?

Rendered shadows and shadow casting detail might be two different things, I thought it is the same.
I don't think the result will be the same with a sun's elevation at 90 degrees. It's not that TG2 thinks "hey sun's right up so no need for shadows"...hence overhangs by the way.

Yes, I'd think geometry is rendered with the MP rasterizer, but it seems some things aren't completely separated.
It seems indeed every element except the colour shaders.
Although I believe the shading of your stones with RDM 1 is just a tad bit smoother, but that might also be the improved geometry.

My guess is, you're right.

Quote from: dandelO on August 10, 2011, 09:52:25 AM
Or, even uncheck 'do ray traced shadows'? Will this then make the displaced surfaces render correctly?

Do you see what I mean, or am I confusing things further?

Yes and no? I don't know! ;D

Title: Re: Terrain and shadows changing during animation
Post by: Oshyan on August 10, 2011, 04:25:09 PM
Only guessing here, but maybe it *is* entirely down to the accuracy of the shadows and inaccurate shadows with lower RDM settings make it *look* ragged, but the micropoly rendered displacement *is* smooth. Hard to test this though because without shadows there's little definition to see whether the edge is smooth. But that's what I would try, disable shadows, leave RDM at .25, and see if you can tell.

- Oshyan
Title: Re: Terrain and shadows changing during animation
Post by: Tangled-Universe on August 10, 2011, 04:32:20 PM
Hmmm I don't know Oshyan. Look at the bottom right part of the fake stone example. Some parts are directly lit and are smoother compared to the example with lowest RDM setting.

Puzzling :)
Title: Re: Terrain and shadows changing during animation
Post by: Oshyan on August 10, 2011, 05:13:47 PM
They all appear to be errant shadows to me. The actual edges (against the base terrain of a different color) appear equally smooth as far as I can see. Which supports my theory. Hopefully Matt can answer definitively.

- Oshyan
Title: Re: Terrain and shadows changing during animation
Post by: Matt on August 10, 2011, 05:21:13 PM
Quote from: dandelO on August 10, 2011, 09:38:39 AM
I said I'd noted this happening on fake stones as well.
Here's two tests, a simple and entirely default fake stones shader, no surface shaders or other modifications added. Both tests at final render detail=1 for consistency. Both with the default micropolygon renderer.

Firstly, an RDM setting of default=0.25 with fully adaptive enabled;
[attachimg=#]

Lastly, the very same with RDM=1;
[attachimg=#]

The difference is plain to see.

I'm out.

Yes, there is a difference, but the difference is caused by shadows (which are ray traced). When you cast shadows from a lower detail version of the geometry, problems like these can occur. They look more pronounced than I expected, and I can look into that to see if I can improve it.
Title: Re: Terrain and shadows changing during animation
Post by: dandelO on August 10, 2011, 05:30:00 PM
**EDIT FOR CLARITY IN MY ABBREVIATIONS; 'RDM'='ray detail multiplier' and 'DRTS'='do ray traced shadows'.**

OK. Here's a quick test I just ran that is pretty conclusive. All at render detail=1, fully adaptive=1, force all edges=1.

It seems that a low RDM *only appears to affect geometry* when 'do ray traced shadows' is checked.

Notice the undefined edge in the first image, I zoomed in very close to the bottom edge of the shape(the single image beneath), where I expected to see the apparent geometry discrepancies magnified. But it didn't turn out that way. The discrepancy is still there but it's shrunk in size, telling me that it is, after all, just stupidly calculated shadows.

The last 3 images in the tall stack picture have identical geometry, I viewed them in sequence and there is no real difference between them at all.

[attachimg=#]

First image viewed close;
[attachimg=#]
Title: Re: Terrain and shadows changing during animation
Post by: dandelO on August 10, 2011, 05:31:17 PM
Matt, you posted before I did, confirming what I just found out the long way! :D

Thanks for the confirmation! :)
Title: Re: Terrain and shadows changing during animation
Post by: Tangled-Universe on August 10, 2011, 05:40:49 PM
Thanks Matt and Martin. Makes more sense now!

Cheers
Title: Re: Terrain and shadows changing during animation
Post by: dandelO on August 10, 2011, 05:41:59 PM
The way to improve it in the meantime, for anyone reading here, is(when not using RTE) set the ray detail multiplier to '1' for correct shadows. Remember, this will multiply render time, especially so in scenes containing transparent water and GI.(and ray traced objects/atmosphere, as well, I'd imagine?).
Title: Re: Terrain and shadows changing during animation
Post by: Matt on August 11, 2011, 12:41:01 AM
Quote from: dandelO on August 10, 2011, 05:41:59 PM
The way to improve it in the meantime, for anyone reading here, is(when not using RTE) set the ray detail multiplier to '1' for correct shadows. Remember, this will multiply render time, especially so in scenes containing transparent water and GI.(and ray traced objects/atmosphere, as well, I'd imagine?).

With shadows, it only affects the shadows of terrain or other displaceable primitives (sphere, lake, plane), but those shadows may be cast onto other things like objects, so it will increase render times all round. Also, ray detail multiplier affects the time taken to render the GI prepass because that pass is entirely ray traced.

Matt
Title: Re: Terrain and shadows changing during animation
Post by: Tangled-Universe on August 11, 2011, 03:36:33 AM
Thanks Matt,

While you're here, can you also tell us about the "force all edges" and "fully adaptive" and in which situations disabling or enabling any of these two might be beneficial?
Thanks :)

Cheers,
Martin
Title: Re: Terrain and shadows changing during animation
Post by: Matt on August 11, 2011, 07:23:19 AM
Quote from: Tangled-Universe on August 11, 2011, 03:36:33 AM
Thanks Matt,

While you're here, can you also tell us about the "force all edges" and "fully adaptive" and in which situations disabling or enabling any of these two might be beneficial?
Thanks :)

Cheers,
Martin

I hadn't documented these settings because they are 'unsupported' settings that may change in future versions. They are supposed to be temporary features that will be replaced by automatic settings in future. The only reason they work in the public release is so that if you are rendering an animation and you have micropolygon popping issues you can follow my recommendations, which are as follows:

For animations:

In the Render Subdiv Settings node:

Fully adaptive OFF, but there may be side effects. OFF may reduce detail unsatisfactorily in some places or put needless detail in others, but if you need absolutely pop-free animation it might need to be OFF in current versions.
Force all edges ON
Jitter shading points OFF
Stabilise ray detail in motion ON (if you have v2.4 alpha)

On the 'Extra' tab of the render node:

Detail blending 1. This is the most important setting for blending between levels of detail as the distance between camera and surface changes. Values other than 1 can be used, but I would not recommend lower than 0.5. Higher blending values increase render time, unfortunately.
Displacement filter 1 (default). Allows displacements to blend between levels of detail. The effect also depends on detail blending.
Microvertex jitter OFF or ON. OFF seems to produce a more stable animation because I think there is an error somewhere, but ON reduces render times by a small amount.
Detail jitter OFF. There is an error that cause this to change from one frame to the next, so you should switch it OFF for animations.

For still images:

Either delete the Subdiv Settings Node altogether, or reset it to default settings, which are:

Fully adaptive ON (default)
Force all edges OFF (default)
Jitter shading points ON (default)
Stabilise ray detail in motion OFF (default)

Still image quality is generally higher and often with shorter render times if you have the following settings on the 'Extra' tab:

Detail blending 0 (default). Higher blending values increase render time, and are not as useful with still images, so I suggest setting to 0. Blending also softens the appearance of surfaces, but its main purpose is to blend between levels of detail in animations.
Displacement filter between 0 and 1. I always leave this at 1 (default) for still and animations.
Microvertex jitter ON (default)
Detail jitter ON (default)

Briefly, this is what the Render Subdiv Settings do:

Fully adaptive causes micropolygons to be more heavily subdivided when the surface is stretched by displacements, but reduces the amount of subdivision where the surface is compressed in screen space due to the angle of view or due to displacements. For stills this is usually a good idea, but it can lead to sudden changes between frames in an animation (and may also produce gaps in stills, although I am not sure about this). Turning this off means that the amount of subdivision is quite regular according to the undisplaced surface, and therefore stable in animations, but doesn't give the best image quality when studying each frame separately. Big displacements will look quite faceted when this is turned off.

Force all edges fixes one of the problems that causes gaps between micropolygons. If two neighbouring micropolygons A and B are subdivided to different levels along a shared edge, this can cause gaps. Force all edges causes the shared edges to be subdivided to the same level. This helps in both animations and stills. The default is OFF because this feature hasn't been in use for very long and it does slightly increase render time. For animations, though, it's probably worth the cost to remove artefacts.

Jitter shading points chooses a random point on each micropolygon as the point where lighting and shaders are calculated. The results of those calculations are used to colour the whole micropolygon. Jittering provides a more natural image, but because of an error in current versions the jitter is different on each frame so it is a source of unwanted noise in animations.

Stabilise ray detail in motion. This feature is even more recent than the others, and needs some further work, but its purpose is to blend between levels of detail when calculating shadows and reflections. If it's working correctly, it should have two benefits. It should minimise the crawling of shadows across surfaces as the distance from camera changes. The crawling motion is instead replaced by a gradual blend between different shadow positions. (Unfortunately it's impossible to completely stop the shadows from changing shape, because the terrain that casts the shadows needs to change levels of detail as the camera moves.) Second, because the shadow positions are now blended instead of moving from frame to frame, popping that occurs when shadow-casting micropolygons very close to the point receiving the shadow change their shape should no longer happen. Unfortunately, stabilise ray detail in motion produces artefacts of its own - sometime a kind of cross-hatching pattern is visible. This is something I still need to improve.

I really don't like having all these options. The goal is hide these behind a couple of simple options which should optimise Terragen for animations or stills.

Matt
Title: Re: Terrain and shadows changing during animation
Post by: dandelO on August 11, 2011, 08:13:57 AM
Thanks very much for those explanations, Matt! :)
Title: Re: Terrain and shadows changing during animation
Post by: MKE on August 11, 2011, 09:20:58 AM
Hi Matt,

thanks a lot for this explanation. This helps very much to understand what's happening and that's way more satisfying than when we just try to turn some knobs, to see what's happening and then interpret what Terragen is doing.

It's also good to know that there are still some bugs in Terragen (and it's not always because I'm to stupid... ;-).

Kind regards,
Martin
Title: Re: Terrain and shadows changing during animation
Post by: Tangled-Universe on August 11, 2011, 09:46:39 AM
Thanks for the extensive explanation Matt.

In the animation section of explanations you didn't really mention the detail blending parameter and displacement filter setting as being important to animation.
Did you forget it or is it on purpose because the "new" parameters overrule the detail blending/displacement filter technically, or by importance?

What about those anyway? I suppose the detail blending parameter blends geometry in between frames to avoid popping geometry, but little clue about the displacement filter?

At the moment I'm working with Jon West (Hetzen) on animating a scene (with vegetation) and are experiencing some issues like popping shadows and especially vegetation flicker.
(the latter I think has little to do with detail blending).

Sorry Martin (MKE) for deviating a bit, but I hope you find this interesting too.

Cheers,
Martin
Title: Re: Terrain and shadows changing during animation
Post by: MKE on August 11, 2011, 10:04:23 AM
Quote from: Tangled-Universe on August 11, 2011, 09:46:39 AM
Sorry Martin (MKE) for deviating a bit, but I hope you find this interesting too.
Sure, still learning a lot from you all... :-)

Martin
Title: Re: Terrain and shadows changing during animation
Post by: Matt on August 11, 2011, 05:29:10 PM
Ah, yes Martin, that was an oversight. I'll edit my post to add those. Thanks.
Title: beton
Post by: Cakbridaroami on August 13, 2011, 12:49:04 AM
yeap (http://siteworld.pl/_/ook.gif)
Title: Re: Terrain and shadows changing during animation
Post by: Tangled-Universe on August 16, 2011, 06:04:21 AM
Quote from: Matt on August 11, 2011, 05:29:10 PM
Ah, yes Martin, that was an oversight. I'll edit my post to add those. Thanks.

Thanks for the update :)
This goes into my archive for reference ;D

Cheers,
Martin