How to improve CPU usage? Tried many bucket settings.

[pixelsmack]

I have tried all sorts of bucket settings and I cannot get Terragen to be more thread efficient. Maybe I am spoiled by Lightwave3Ds thread efficiency but this is rather poor. My Tgen renders slam the CPU at 100% for the first 85% of the frame. Then as it gets down to the final sections, cpu usage drops from 100% to 12% or so. Wow. this makes a frames that should complete in 2-minutes, take 15-minutes.

What's going on?

AMD 3960X, Windows 10 Pro.

[WAS]

How many threads do you have? I see only 6 buckets going which would be 6 threads. For my CPU that would be half load. This is natural as TG doesn't split jp remaining buckets when it has idle threads. Though this may be a feature planned. Would be nice if it goes to do a new bucket and sees there is one thread idle, it splits that bucket, and if another bucket starts and 2 are idle, split the bucket 3 ways, etc. This would finish the rest of the scene faster than just counting down buckets active and leaving the rest idle.

[Kadri]

Yeah i use Lightwave too and its adaptive rendering is sweet.
Unfortunately Terragen doesn't have that for now at least, as Jordan said above.
You could render with smaller bucket sizes to overcome this to a degree as you tried.
Default is 256 i think. I use mostly 128. Lower then 128 gets diminishing return in render time.
I don't go lower then 96 (not sure what Matt would say about this).
Another approach could be rendering these kind of scenes upside down.

[WAS]

Also depending on scene settings you could crop render the problematic area and comp

[digitalguru]

Your CPU usage will reduce as the render completes as you show in your image, so that's to be expected - it's only rendering the last few buckets, and as the thread count reduces, so does the CPU load.

There was a thread a while ago about optimal core/thread count for Terragen, I think it might have been 16 cores, and above that the law of diminishing returns kicks in.
We're getting many more cores in modern CPUs but not much software can fully take advantage.

[D.A. Bentley (SuddenPlanet)]

Kadri has the right idea to render your scene upside down, so the water which takes longer will start first.
Just rotate your camera 180 in Z.
If your doing an animation it might be a little too much extra work, but for stills it works great, then you just flip the frame back in Photoshop.

[WAS]

I wonder how much this actually helps. Ive done this and noticed same buckets stuck slowed down. The only difference is some buckets can continue on, at a slower pace for overall render. I haven't noticed much time really saved. Same slow down in calculations is happening, and rest of image rendering with crippled threads rather than just finishing the problematic last threads, which are now first and still calculating at same speed while actual render progress with fewer buckets is slowed

[Kadri]

I wonder how much this actually helps. Ive done this and noticed same buckets stuck slowed down. The only difference is some buckets can continue on, at a slower pace for overall render. I haven't noticed much time really saved. Same slow down in calculations is happening, and rest of image rendering with crippled threads rather than just finishing the problematic last threads, which are now first and still calculating at same speed while actual render progress with fewer buckets is slowed

It is very scene dependant. And i only bother with it in animations.
The best method i found is to render 2 or even 3 instances with Terragen to overcome some extreme cases.
Each instance takes longer to render. But overall i got faster render times per frames in general without fiddling with affinity.
But this needs more RAM too of course. This was what i got but can't say that it works all the time for sure.

[D.A. Bentley (SuddenPlanet)]

The typical type of scene I render that I see a huge difference is very large panoramas (ie 8k Spherical Renders) where there is water.  The reason it is faster is because the water area can start rendering from the very beginning so even if there are some very slow buckets, they don't end up at the end of your render hanging up the render completion with just one or two threads being utilized.  I hope that makes sense.

Derek

[WAS]

I still would like to see some tests. To my logic doing similar intensive stuff with languages (not rendering) you kinda want to do the fast stuff first when multi-tasking, cause you end up with threads lagging behind on larger tasks which reduce overall performance. Multi-threaded file management comes to mind, I always organize by smallest size and assign groups accordingly.

[Kadri]

I  think it depends on the scene, where the slow buckets are, how many they are... I made a very basic test with normal and flipped camera.
Normal took 4.15 minutes, 180 degree rotated 3.56.

This time difference could be important with higher and complicated settings, resolution etc.
I will make some more tests.

[Kadri]

Same scene with higher settings:

Normal : 49.51 minutes
180 degree rotated camera:  45.13 minutes

This was just a random scene i made.

I will test my real animation scene i am rendering right now.
It does have more going on on the lower side too.
But not sure if there will be any difference.

[Kadri]

The animation scene i tested:

Normal: 23.42
180 degree rotated camera: 23.07

The difference is small here but it is really scene dependant as even the rotated render wasn't quite efficient.
It rendered for some minutes for only 50% and even much much lower cores and still was faster.
Even this small difference makes nearly 1 day of rendering difference for 2000 frames.
But as i did the rotated test after the normal one i am not sure from where the difference really comes as the difference isn't so high (hard disc or  RAM caching etc.).

With this difference i won't bother. But with a scene that does have high displacement, reflection etc. problematic long bucket renderings it could make a huge difference.

Just got curious if Matt could make the bucket rendering optional. From bottom to top or like in Cinebench spiral etc.
Or just a basic option to rotate the camera internally while rendering.
Rotating the camera manually isn't hard but with an animated camera it brings an unneeded error possibility for users.

[D.A. Bentley (SuddenPlanet)]

The idea I presented to Matt a year or two ago was to incorporate a predictive pre-pass into the Terragen render engine to determine which buckets would potentially take the longest, and based on the prepass results let Terragen decide the best bucket order, and if there are available cores split up buckets into smaller buckets (So your not stuck with just one bucket size).

This could be an option to toggle, so for quick previews renders you could turn it off because it would likely not be needed with low to medium quality settings.  It would be useful for high resolution (8k+) still renders though.

A manual method for doing this is to figure out on your own the problem areas, and do multiple crop renders with optimized render/bucket settings and then stitch the final result.

[Kadri]

Yeah. Adaptive rendering would be the best. But i lost some of my enthusiasm-expectation for bigger changes (sorry Matt).
So i talk more about less hard to do (for me of course) things in general.