problem with CPU % usage

Started by Frank_Art, October 23, 2018, 11:27:31 AM

Previous topic - Next topic

Frank_Art

Hi, sometimes TG is not able to use all the CPU core, when it start is like 95% then it drop to like 16% CPU usage and it go up and to 35% and it keep float up and down at those level of usage until it finish to render the image.
I'm rendering mostly clouds with a 1950x (threadripper 16/32) and I also have 2x 1080ti SLI (i know they are useless for TG but maybe there is some driver conflict)
On advanced setting I leve 256x256 Max bucket size, check "Allow auto reduction" and put 32 Maximum threads.
Some idea why this is happening? there is any way to let G use 100%CPU constantly?
Thanks

WAS

While you're waiting for a reply, refer to this topic that was just had recently here in support: https://planetside.co.uk/forums/index.php/topic,25708.0.html

Frank_Art

thanks, i read it pretty fast not sure I get what it was his problem, in my case in first instance can be the GI pass going through 9 Cloud layer v3 and in the end it can be that the size of the image is not a multiplayer of 256 (size of the bucket) and TG is not able to find a minor common denominator in manner to use all the 32 core concurrently or something like that... is it possible in your experience?
cheers

WAS

Quote from: Frank_Art on October 23, 2018, 04:17:07 PM
thanks, i read it pretty fast not sure I get what it was his problem, in my case in first instance can be the GI pass going through 9 Cloud layer v3 and in the end it can be that the size of the image is not a multiplayer of 256 (size of the bucket) and TG is not able to find a minor common denominator in manner to use all the 32 core concurrently or something like that... is it possible in your experience?
cheers

Yes, and the prepass is single-threaded, I'm pretty sure at least.

Oshyan

Prepass is not single threaded, but I've found it can sometimes be less multithreading-friendly than the primary render pass, i.e. lower CPU usage during that pass (but not single thread).

9 cloud layers is A LOT, and I am not surprised if it reduces efficiency. It will also likely take a lot of memory, so make sure you're not running into memory issues.

You can also try reducing the bucket size to see if it has an effect, positive or negative.

Other than that I'm not sure why your CPU usage at 32 threads would not be averaging in the high 90s, although I do know that Threadripper has some efficiency issues due to memory access/speed restrictions in the current architecture (or something along those lines).

- Oshyan

Frank_Art

Thank's good to know, I dont think memory is a problem, the scene is taking around 10 GB and I have 64 so... I'm gonna try a smaller bucket and set the image resolution as a multiplayer of that size to see if there is any improvement, if not I think is time to train patience ^^
Cheers

WAS

You may also find some interest in this educational example I put up about layering clouds with a single cloud layer. It can be a little bit of a pain to work with but can achieve some beautiful effects with serious optimization utilizing a single cloud layer (clouds take forever for me and this method is seriously faster than just one more cloud layer).

https://planetside.co.uk/forums/index.php/topic,24581.0.html

Quote from: Oshyan on October 23, 2018, 05:07:10 PM
Prepass is not single threaded, but I've found it can sometimes be less multithreading-friendly than the primary render pass, i.e. lower CPU usage during that pass (but not single thread).

Oh, I must be thinking of something else. I remember Matt telling me something I was doing and thinking was strangely slow was single threaded.

Frank_Art

Thanks alot WASasquatch, I'm gonna give it a check for sure ;) Unfortunately I can not share what I'm working on because I'm under NDA but I can say that I'm doing a "Hero epic alien storm" with specific shapes and volumes, creating tons of clouds layer allow me to brake down the storm into module parts and have full control of the shape and position in space. Right now I need to find a way to render them in "reasonable time".
Last night I found that having the image a multiple of 256 can help and also the problem about the CPU% is only for the first passes but since I have tons of layer it can take alots lots of time... do you know if there is any way I can render for example 3 clouds layers keeping the others 6 clouds layers transparent (but they effect shadows) in manner to split the final image in 3 render parts? dose it make sense?
Cheers

WAS

Quote from: Frank_Art on October 24, 2018, 10:24:28 AM
Thanks alot WASasquatch, I'm gonna give it a check for sure ;) Unfortunately I can not share what I'm working on because I'm under NDA but I can say that I'm doing a "Hero epic alien storm" with specific shapes and volumes, creating tons of clouds layer allow me to brake down the storm into module parts and have full control of the shape and position in space. Right now I need to find a way to render them in "reasonable time".
Last night I found that having the image a multiple of 256 can help and also the problem about the CPU% is only for the first passes but since I have tons of layer it can take alots lots of time... do you know if there is any way I can render for example 3 clouds layers keeping the others 6 clouds layers transparent (but they effect shadows) in manner to split the final image in 3 render parts? dose it make sense?
Cheers

With the layering method you have the same control with surface layers and masks, but the benefit of a single cloud layer driving it. I wish you luck on your project.

Oshyan

Frank, is the project something you could share privately to Planetside Software support? I'd be happy to take a look and see if I can help you optimize the render time.

I believe one method for getting clouds onto separate render passes would be to use the Render Layer functionality and *separate planets* with the same position and all but the first planet having the surface set to not render. Attach the cloud layers you want to separate onto the other planets, and then you can specify those planets as the objects to control the Layer separation. You'd need to do 3 separate renders though, you can't currently output the layers separated but in a single render pass.

WAS, all post processing is single threaded, so the part of GISD that applies at the end, or the glows, etc. Those can be slow at the end, especially on high resolution renders.

- Oshyan

Frank_Art

#10
Thanks Oshyan, I will write you in PM . I need to try to split the scene on separate planets, that's the missing part :)

I would like to know  are the most affluent factor for a clean sky/clouds render and  one are effecting the render time alot?  personally what I change are those parameter, then you guys can maybe tell me if it's right or not, maybe there are other setting to keep in consideration.

resolution spherical: 4096 x 2048 (I personally would like to double it but if I do I need to sacrifice some other quality setting I guess)
Micro Detail 0.6
Anti-aliasing 6
GI in clouds: Still/Very high
Ray-marching quality: 1 (in each cloud layer)
Do soft shadows: Activate

Cheers

Oshyan

Micropoly detail doesn't matter if you're not rendering the terrain *and* you're using Defer Atmo (which you should be).
AA of 6 could be fine, the the real key is the adaptive sampling and the Noise Threshold.
For GI in clouds, did you test with lower settings? Unless you are animating, you may not need it that high.

The other thing you should look at is the Voxel Scattering Quality in the GI in Clouds tab of the GI Settings window (accessed from Quality tab of Render node). Default value should now be 100, though yours may be at 50 or something else. Try 100, and try higher as well, depending on your AA adaptivity settings, and cloud quality settings. You may find a value of 100 here, with lower cloud quality (e.g. 0.5) works better, for example.

There are a lot of settings to test, so ideally we can just see the TGD and make more direct suggestions based on actual tests.

- Oshyan