Cloud tests

Started by RogueNZ, October 13, 2019, 12:11:12 AM

Previous topic - Next topic

Dune

I believe it hung halfway, as seen on his half render image a few posts up.

RogueNZ

Correct, it's hanging part way through renders. I was able to render cropped sections (e.g. one for the foreground DoF effect, another for the soft shadows on the windmill), but seems to be when combining some of these aspects that perhaps exponentially increase render times. It crawls down and eventually stops as it gets towards the base of the cloud. The issue was present before I added the second rain cloud layer, but doing that definitely increased render times in that region of the image significantly (more than double).

I'm not 100% sure it is actually hanging, it could still be doing something, but the fact that it will render to the base of the clouds in three hours, but no further progress for at least two or three, makes me think it has. Any tips for optimizing this scene would be appreciated!

WAS

Quote from: Oshyan on October 20, 2019, 12:07:44 AMFrom his report it sounds like it's hanging at the *end* of the render, so not necessarily during cloud calculation, but more likely a post process effect of some kind. His scene has all of them enabled, each of which is going to use more memory at the end of the render I believe. Many of these may also be single-thread effects, not certain about that though.

- Oshyan


Only one way to find that out for sure, and that's a windows error event. It'll log it. That should be one of the first steps to diagnosing crashes to see if it's actually RAM related.

Oshyan

It's not actually crashing from the sound of it, so no Windows error is likely being reported.

- Oshyan

Oshyan

OK so I did a series of tests here, I wanted to get as much info as I could before replying.

I did end up being able to finish the render successfully. There is one caveat as far as this being a totally accurate test, which is that the objects weren't included, and though I located the same objects I already had, the textures apparently didn't load properly, so my grasses are all textureless and black. :D I can redo the test(s) again, but I'll note a few things.

First of all the total render time was nearly 20hrs. I limited it to 16 threads to try to more accurately reproduce the memory use you would be seeing on your machine. I have 32 threads available so I could redo it and might see 20% better performance, perhaps a tad more, since all my *physical* cores were theoretically in use in the last render. But the point is it's a pretty long render time.

Second and perhaps most usefully, the render did seem to "hang" after completing a single bucket on the right side of the lower part of the scene. I didn't see any other buckets appearing to do work once that one was finished, which was the main thing that seemed unusual, however this may have mainly been because the ground area is rather dark, so you don't see the in-progress buckets very well. That being said the lower third of the image did take a surprising amount of time relative to the upper 2/3, easily half the render time. With path tracing it's not unusual for the ground to take longer than clouds, but it still was a bit surprising given how little area of the image the ground takes in this case. And in fact I don't think it was the ground so much as the *horizon* that took a long time. I suspect, actually, that it's the lower part of the cloud rather than the distant vegetation, and this could be due to the complex cloud setup you have going on in this scene.

Anyway, without further testing I'd say it's just doing a lot of calculation (perhaps some unnecessarily) in the lower part of the cloud and/or distant horizon. This suggests that optimization may be the best option. Ideally you might find a way to optimize your creation of the cloud's shape with fewer nodes, especially whatever you're doing that affects the bottom area of the cloud. But that can be a long process. Render optimization is easier to apply and I'd say there are a relatively few things you'd want to adjust to get a much faster result for equivalent quality.

The first would be to reduce Micropoly Detail from the high level of 0.8 to 0.5 or even lower, since most (if not all) of your micropoly-rendered surfaces are covered by ray traced objects. This may not have a huge effect because of that, but it's needless calculation in any case, so worth reducing. The other most important thing if you have 4.3 or newer (and especially in 4.4) would be to enable the Robust Adaptive Sampler and go with the defaults at 1/64. I'd also recommend increasing AA to 8 just so your minimum samples ends up at 1 sample at 1/64th. Increase the Voxel Scattering Quality (GI in Clouds tab of GI settings) to 2-400 (I used 400), and also the cloud quality to 1. Then do some crop renders and reduce the Pixel Noise Threshold until the noise level is acceptable to you. I settled on 0.025, which is unusually low actually, but was needed to get equivalent noise level in this case, vs. your original.

In a 16 thread test of those changes here, render time decreased dramatically to a little less than 7hrs, about 1/3 the total render time. Not bad!

So ultimately I don't think there is any crash or even memory-related problem here. I think it's just an unusually intensive part of the scene that makes it appear that the render is hung. I think/hope that if you left it to run for ~20hrs or so, it would have finished. But try the optimization ideas above and see if you can get the scene to render as originally configured, but in a more reasonable amount of time.

- Oshyan

RogueNZ

Sorry for the delay replying, I did read this and some of your suggestions came in handy for my latest scene, the time saving has been a huge benefit! Shows how much I still have to learn about these settings.

I also agree that it is likely something to do with the base of the cloud, perhaps the flatness of it and the camera being almost parallel might be causing some interesting ray casting situations? I definitely could have put some more effort into optimizing the node network as well.

Appreciate the effort you put in here Oshyan, cheers.