surface layers and path tracing

Started by KyL, January 10, 2020, 10:43:18 am

Previous topic - Next topic

KyL

January 10, 2020, 10:43:18 am Last Edit: January 10, 2020, 10:52:24 am by KyL
Hey there,

I am noticing quite a big change in rendertime when using the path tracer with stacked surface layers.
I took the default scene and stacked 4 surface layers with simple color and displacement. The rendertime were are follows:

standard renderer:
-default: 14s
+1st layer: 15s
+2nd layer: 16s
+3rd layer: 17s
+fake stones: 17s

render time difference: +21%

path tracer:
-default: 17s
+1st layer: 22s
+2nd layer: 23s
+3rd layer: 28s
+fake stones: 38s

render time difference: +123%

On more complex scene this becomes even more noticeable, I see changes of several hours between the standard and path traced version. I imagine this being inherent to the way both renderer evaluate the shading network. I am not sure if there is a miracle solution here, I am thinking about using merge shaders instead of surface layer and see if this helps.

Matt

Hi KyL,

Do you find this to be the case when the surface layers generate only colour? Are you using fractals to generate colour and/or displacement?

Unfortunately the path tracer needs to evaluate the surface colour and any other BRDF attributes generated by the shader at each bounce. If these are tied to complex functions such as fractals or Voronoi functions, this can end up being a large contributor to the render time. Displacements also need to be computed multiple times for each ray. The Subdiv Cache helps when rays hit quite close to the point they originate from, but for long rays the cache becomes ineffective and displacement shaders can end up being a large contributor to render time.

The Standard Renderer can also be affected by this when you ray trace reflections and shadows. But in the path tracer these problems affect GI as well, so they have a larger impact.

I will create an option to use the GI Cache at high ray depths (optionally restricted to long range bounces). For scenes with complex shaders it will probably be much faster to render this way. If it's restricted to long range bounces, not short, it should not have much negative effect on the image quality.
Just because milk is white doesn't mean that clouds are made of milk.

KyL

This was indeed using fractal to drive color and displacement.

Quote from: Matt on January 10, 2020, 11:17:32 amthe path tracer needs to evaluate the surface colour and any other BRDF attributes generated by the shader at each bounce.
Thanks for confirming that. I feel like most of the computation time comes from multiple displacement evaluation, which indeed might not be needed in all cases. I usually set up Ray Detail Multiplier to 0.5 as it makes quite a difference in shadows, but I overlooked that with the path tracer it will affect all other rays as well.



Quote from: Matt on January 10, 2020, 11:17:32 amI will create an option to use the GI Cache at high ray depths
This sounds like a great idea. For many things, especially landscape, we might not need to trace accurately along the whole ray distance/bounce. 


Thanks!

KyL

Quick update on this. I kept working on the scene where I noticed the biggest time differences between standard and the path tracer.
Originally I couldn't get a path-traced frame at all, as I had render times over 15 hours for one single frame. This made me kick off the little experiment above to flag a potential issue.

As I worked more on the scene and refined my surfaces and displacements, I did another try with the path tracer. I set the max depth to 2 (landscape only, shouldn't need more) and the max subdiv cache to 32Gb (way too much but I have the RAM so the hell with it)
The first test frame rendered in 41min, against 24min with the default renderer. This is much more in line of what I expected to see, perhaps even faster considering I have motion blur on. So great results here!

I probably had a lot of unneeded reflections going on: killing the specular contribution on a sand surface layer might have helped as well!

WAS

Quote from: KyL on January 13, 2020, 11:49:49 amQuick update on this. I kept working on the scene where I noticed the biggest time differences between standard and the path tracer.
Originally I couldn't get a path-traced frame at all, as I had render times over 15 hours for one single frame. This made me kick off the little experiment above to flag a potential issue.

As I worked more on the scene and refined my surfaces and displacements, I did another try with the path tracer. I set the max depth to 2 (landscape only, shouldn't need more) and the max subdiv cache to 32Gb (way too much but I have the RAM so the hell with it)
The first test frame rendered in 41min, against 24min with the default renderer. This is much more in line of what I expected to see, perhaps even faster considering I have motion blur on. So great results here!

I probably had a lot of unneeded reflections going on: killing the specular contribution on a sand surface layer might have helped as well!

Ice noticed reflections due heavily impact PT, but that is probably to be expected scattering paths.

KyL

I have further feedback on this. 
I tried to isolate the reason why I was getting such rendertime differences. 
It seems that "intersect underlying" option of the surface shader might be a culprit, but I am not 100% sure. The thing I know for sure it that the issue in rendertime is due to CPU under-utilization. 
For example a standard render will use the CPU at 100%, but switching to path tracer will drop to 15% after a few minutes. I do not know the cause for sure and tried to isolate it but I haven't been successful so far. I tried a different size cache, bucket sizes, preallocate cache, no success. I have the feeling that switching on and off this surface layer with "intersect underlying" somehow breaks the full CPU use.  :-\

WAS

Quote from: WAS on January 13, 2020, 02:22:40 pm
Quote from: KyL on January 13, 2020, 11:49:49 amQuick update on this. I kept working on the scene where I noticed the biggest time differences between standard and the path tracer.
Originally I couldn't get a path-traced frame at all, as I had render times over 15 hours for one single frame. This made me kick off the little experiment above to flag a potential issue.

As I worked more on the scene and refined my surfaces and displacements, I did another try with the path tracer. I set the max depth to 2 (landscape only, shouldn't need more) and the max subdiv cache to 32Gb (way too much but I have the RAM so the hell with it)
The first test frame rendered in 41min, against 24min with the default renderer. This is much more in line of what I expected to see, perhaps even faster considering I have motion blur on. So great results here!

I probably had a lot of unneeded reflections going on: killing the specular contribution on a sand surface layer might have helped as well!

Ice noticed reflections due heavily impact PT, but that is probably to be expected scattering paths.
Quote from: KyL on January 16, 2020, 06:03:07 pmI have further feedback on this.
I tried to isolate the reason why I was getting such rendertime differences.
It seems that "intersect underlying" option of the surface shader might be a culprit, but I am not 100% sure. The thing I know for sure it that the issue in rendertime is due to CPU under-utilization.
For example a standard render will use the CPU at 100%, but switching to path tracer will drop to 15% after a few minutes. I do not know the cause for sure and tried to isolate it but I haven't been successful so far. I tried a different size cache, bucket sizes, preallocate cache, no success. I have the feeling that switching on and off this surface layer with "intersect underlying" somehow breaks the full CPU use.  :-\
I've mentioned this a few times. PT under utilizes CPU. Buuut I also get strong CPU hiccups with PT compared to standard utilizing 100%. I don't get it. By hiccups I mean complete system lockups for a moment. Usually with some sort of intense PT aspect like reflections or SSS. Normal hard surfaces not so much (as far as lockups whole under-utilized)

WAS

I'm actually curious to test a 4.3 or 4.1 on my Linux server too cause no matter what the second CPU only uses 45-50% with peaks to 75%, whole first CPU will be 100 across all cores. Doesn't seem normal and wonder if it's always been this way.

On my old renderfarm before they dropped TG I often got few dollar overcharges that made no sense from info I gave them on muuuuuch lower end system. Wonder if that was why. Scenes dragged out longer and used more power than they should have based on my calcs on windows.

KyL

Interesting feedback. I haven't tried on previous build, but what I did try is reducing the number of threads. Rendering on 16 threads displayed the same behavior. It was way below the supposed 50% CPU capacity.

I am still trying to figure out what causes the slowdown but it's really time consuming...

Matt

I'll take a look at the scene if you want to package it up and send it to me. I definitely want to fix those sorts of bottlenecks.
Just because milk is white doesn't mean that clouds are made of milk.

KyL

Thanks Matt. 

I managed to fix the issue by replacing surface layers by merge shaders inside a procedural dunes setup. It might be related with surface layers being nested inside other another surface layer, even though it is the first time this causes me trouble.

I will see if I can simplify the scene into a more "debug ready" version and send it over.

WAS

Quote from: KyL on January 17, 2020, 02:34:07 pmThanks Matt.

I managed to fix the issue by replacing surface layers by merge shaders inside a procedural dunes setup. It might be related with surface layers being nested inside other another surface layer, even though it is the first time this causes me trouble.

I will see if I can simplify the scene into a more "debug ready" version and send it over.

Definitely interesting considering in the past, and even currently, Merge Shaders are heavier. For example, merging displacement or fake stones is much faster with surface layers.

Definitely seems like something isn't right. Were you using better displacement continuity or colour continuity?

KyL

Actually I do use merge shader to blend fake stones with "choose by altitude: highest". It is the fastest way I found to avoid fakestones growing over fakestones.

My "fixed" frame renderered fine and Terragen froze at the very end, never came back. So something is definitely wrong with threads crashing either during or at the end of the render. I don't use the better displcement or colour option though.

WAS

Quote from: KyL on January 17, 2020, 03:09:20 pmActually I do use merge shader to blend fake stones with "choose by altitude: highest". It is the fastest way I found to avoid fakestones growing over fakestones.

My "fixed" frame renderered fine and Terragen froze at the very end, never came back. So something is definitely wrong with threads crashing either during or at the end of the render. I don't use the better displcement or colour option though.

Hmm if your not using the new surface layer options than it may not be those relatively recent additions. Very strange.