183 hours and counting - Did I over do my render settings?

Started by D.A. Bentley (SuddenPlanet), June 10, 2019, 10:55:07 PM

Previous topic - Next topic

D.A. Bentley (SuddenPlanet)

I can't show anything at the moment, but I'm doing an 8k Skybox render on a 12-core Xeon and have plenty of RAM (64GB) and the render I'm doing has been going for 183 hours so far, and based on it being about 2/3rds done I estimate it will need another 90+ hours to finish.  Not sure if I overdid my render settings, but I'll post some of my settings to see if I'm way off on something.

Render Quality
Micropoly detail = 0.8
Anti-aliasing = 8
Defer atmo/cloud = On

Sampling:
Robust adaptive sampler (BETA) = On
Customise sampling = On
First sampling level = 1/64
Pixel noise threshold = 0.0125

GI Settings:
Prepass Tab:
GI cache detail = 2
GI sample quality = 2
GI blur radius = 8
Supersample prepass = On
Image Pass Tab:
GI surface details = On (All Defaults, 1,1,24)
Exaggerate surface details = On (All defaults, 1,1,24)
GI in Clouds Tab:
Cloud GI quality = Still / Medium

Other Info:
Image width = 8,000 x 4,000
Soft Shadows = On (diameter = 4, samples = 18)
Atmosphere samples = 16

I have 5 Cloud layers, all are Cloud layer v2, and they range from .002 sharpness up to 32.
All 5 cloud layers have a Quality setting of 1.0 and the Acceleration cache is set to None on all.

There is also a Terrain rendering in this image with some textures, tree models, and Daniil erosion.

I'm guessing I set the pixel noise threshold too fine (0.0125) but I have God Rays in this render and felt like going to a bigger number would compromise quality.  I don't want to kill the render and retry the settings because I've already got too many hours invested in this render.  I wish there was a way to save the render progress in these types of situations, because if I had a power outage I would lose all the render progress.  Maybe I'm hinting at a feature request.  ;)

Anyways when this render finishes I can try some setting adjustments.

Thanks,

Derek

Oshyan

I would agree that the Pixel Noise Threshold (PNT) is quite low in this case and is probably contributing to the very high render times. With Robust Adaptive you typically actually need to *increase* the threshold to match non-robust levels of quality (and it will achieve that at a generally lower render time). I'd also use a higher level of adaptivity with AA8, probably 1/256. Robust Adaptive performs best when there is more adaptivity it can work with. 5 cloud layers is also just quite demanding in general.

What is your Voxel Scattering Quality set to (in Cloud GI settings), and are you using 4.3 or 4.4?

- Oshyan

D.A. Bentley (SuddenPlanet)

I'm using Terragen 4.4.18

Voxel scattering quality = 50
Cloud GI may ray depth = 2

Oshyan

OK. So in the future I'd recommend a higher PNT and Voxel Scatter Quality of 100 as a baseline (in 4.4 100 should be the default, so I'm assuming this is an older scene). If you have noise *in clouds*, but nowhere else, then try a higher Voxel Scatter Quality, 200 is a good value.

If your noise is in godrays that's usually the atmosphere and it's generally better to selectively target that rather than lower the PNT for everything, i.e. increase Atmo Samples a bit (24 for example, not a lot). Put another way, in general you want to try to directly increase the sampling quality of whatever element is currently noisy, i.e. undersampled. If your whole scene - or most of it - is noisy, then yes it makes sense to lower the overall PNT, otherwise try to deal with each noisy element on its own where possible.

- Oshyan

Dune

That's extremely long indeed. You also have 18 soft shadow samples, which is a lot! Do you really need soft shadows for the terrain? I often even decrease number of samples if, say, there's a lot of foliage, which will 'soften' shadows anyway, and make any noise less visible.
Maybe doing such big renders is better done in 2-4 big crops, with a GI cache made in advance. I can feel your anxiety about power failure; I have that too when doing large pieces.
Hope it works out after your expectations!

Oshyan

Ah, yes, didn't notice the 18 soft shadow samples, although with a diameter of 4 it is more necessary sometimes. Personally in this case I would enable Defer All Shading and then use like 2-4 soft shadow samples. With Defer All you can use far fewer samples. Also/alternatively consider turning off sample jitter for soft shadows and using fewer samples. This can cause banding in some cases, but often is not noticeable and is faster to render.

- Oshyan

WAS

#6
Dereck, I see you also have Exaggerate Surface Detail on, which means you are Path Tracing?

On my system the PT will slow down and not use all of my CPU (80-90%) when not even half the image is done and 4 buckets hard at work. Perhaps you're having bucket slow-downs on-top of everything. PT just takes longer to render, often double or triple times, which I think may be because I'm not seeing 100% load with my fans firing on full blast the whole time.

This is kinda a good thing though, as when rendering v3 clouds in standard renderer, my system will say 100% load, but seems to be using more load than 100% and temp exceedes even CPU stress tests (even leaving the stress test going for over an hour to achieve max temps) I believe Danny has seen this weird load limit issue as well.

Oshyan

You can have Exaggerate Detail on and still not be using PT - it just has no effect.

- Oshyan

WAS

#8
Quote from: Oshyan on June 11, 2019, 01:41:33 PM
You can have Exaggerate Detail on and still not be using PT - it just has no effect.

- Oshyan

I am assuming he knows this, and is using it correctly, of course.  L.O.L

Curiously, how is the feature ignored? Ignored once before project start, or per bucket? Could add hiccups.

D.A. Bentley (SuddenPlanet)

Thanks Oshyan & Dune!  I can't wait to try these suggestions later on Friday when my render finishes.  ;)

I was in a hurry to leave on my vacation last week, so I started the render without testing the quality settings and was surprised when I came back 3 days later the render was still going.  haha!  Oh well, I am learning a lot.

WASasquatch, I am not using the path tracing renderer.  It's just that those were turned on when I was testing, but having that turned on while using the standard renderer shouldn't affect anything from what I understand.

-Derek

WAS

Quote from: D.A. Bentley on June 11, 2019, 01:44:33 PM
Thanks Oshyan & Dune!  I can't wait to try these suggestions later on Friday when my render finishes.  ;)

I was in a hurry to leave on my vacation last week, so I started the render without testing the quality settings and was surprised when I came back 3 days later the render was still going.  haha!  Oh well, I am learning a lot.

WASasquatch, I am not using the path tracing renderer.  It's just that those were turned on when I was testing, but having that turned on while using the standard renderer shouldn't affect anything from what I understand.

-Derek

It shouldn't, but all depends how it's ignoring it really. It could add to computation hiccups if it's a constant check per-bucket while 100% load, but I doubt that is the case.

But curiously, this is mostly clouds with like terrain below? Are you getting 100% load on your CPU? Buckets can slowdown in standard renderer too but usually related to not much work having to be done in that specific bucket, but the next bucket won't start until it's finished (or another finishes). Could just be seeing lazy buckets finish adding greatly to render times.

D.A. Bentley (SuddenPlanet)

#11
QuoteBut curiously, this is mostly clouds with like terrain below? Are you getting 100% load on your CPU? Buckets can slowdown in standard renderer too but usually related to not much work having to be done in that specific bucket, but the next bucket won't start until it's finished (or another finishes). Could just be seeing lazy buckets finish adding greatly to render times.

WAS, the render is an 8k skybox render so thats using the spherical camera at a 2:1 aspect ratio (8000 x 4000 pixels) and I have multiple cloud layers in the upper half of the image and one low laying cloud layer above the terrain.  The lower layer is actually touching the terrain like low laying fog/clouds.  The terrain has a high detail Classic Erosion on it (16k) and on top of that there are a few xFrog tree populations as well as textures and masks.  The render itself is using about 50GB of the 64GB I have in that 12-core Xeon, and the CPU is constantly 99% - 100% pegged.  I usually only see the CPU utilization go down at the very end of the render when the very last buckets are finishing.  I wish the renderer had some tech that would use all of the CPU even at the end.  I thought thats what Automatic Bucket size was suposed to do, but I never see buckets decreasing in size after the render has started.

Luckily I have three 1500VA UPS's protecting from power loss and no thunder storms are in the forecast so I should be good.  I also forgot about the Render Pause button, so If there was a power loss I could always pause the render to conserve power, and turn off my monitors.

When the game I'm working on is released I can show the render, but for now I can only give a description.

-Derek

WAS

Quote from: D.A. Bentley on June 11, 2019, 02:31:43 PM
QuoteBut curiously, this is mostly clouds with like terrain below? Are you getting 100% load on your CPU? Buckets can slowdown in standard renderer too but usually related to not much work having to be done in that specific bucket, but the next bucket won't start until it's finished (or another finishes). Could just be seeing lazy buckets finish adding greatly to render times.

WAS, the render is an 8k skybox render so thats using the spherical camera at a 2:1 aspect ratio (8000 x 4000 pixels) and I have multiple cloud layers in the upper half of the image and one low laying cloud layer above the terrain.  The lower layer is actually touching the terrain like low laying fog/clouds.  The terrain has a high detail Classic Erosion on it (16k) and on top of that there are a few xFrog tree populations as well as textures and masks.  The render itself is using about 50GB of the 64GB I have in that 12-core Xeon, and the CPU is constantly 99% - 100% pegged.  I usually only see the CPU utilization go down at the very end of the render when the very last buckets are finishing.  I wish the renderer had some tech that would use all of the CPU even at the end.  I thought thats what Automatic Bucket size was suposed to do, but I never see buckets decreasing in size after the render has started.

Luckily I have three 1500VA UPS's protecting from power loss and no thunder storms are in the forecast so I should be good.  I also forgot about the Render Pause button, so If there was a power loss I could always pause the render to conserve power, and turn off my monitors.

When the game I'm working on is released I can show the render, but for now I can only give a description.

-Derek

Since you're using Classic Erosion, despite detail, the terrain should be rendering fairly quickly since it's a heightmap with shading. I can do a full (freeware) render with Classic Erosion and shading in 5 minutes at 1280x720 with upped GI.

Clouds are probably the main issue combined with settings, like the shadow sampling. Especially cloud intersection zones if v3 is used.

Oshyan

Forgot to mention, if using Defer All (which I'd probably recommend), you could also reduce Micropoly Detail to probably 0.5 or 0.6.

- Oshyan

D.A. Bentley (SuddenPlanet)

I tried Defer All on another scene to see how it works, and it makes the render look like it does when Path Tracing is on (All that weird orange & red stuff).  Why is that?

[attachimg=1]