Path Tracing Slowdown

Started by WAS, July 17, 2019, 12:56:23 PM

Previous topic - Next topic

WAS

Still having issues with the path tracer sometimes using no more then 20% of the CPU.

Last time I was going to render my aluminum balls. And so I started, it, whatever, it finished 1 row and 4 buckets in 18 minutes while I went to the bathroom before bed. When I came back I was like "Wow, it's going to probably take longer with the sky"... So all I did was raise the camera, and tilt it down at the balls so only the ground and balls were in the shot. I clicked render and went to bed.

In the morning, there was only one row and 6 buckets done... in 8 hours and 45 minutes... since it was 18 minutes for slightly less, that's like what? a 10000x increase in render time for nothing but a camera adjustment?

What gives? Nothing changed in the scene but camera angle. And again, the CPU wasn't doing anything really (it was literally at 24% when I stopped the render) and all fans were off and computer cooled like it has been like this all night. What could be doing this? This seems to happen to me all the time when I Path Trace where I just waste an entire night to nothing.

I mentioned this with the Benchmark where I got significantly longer render times on a second render.  I'm on Terragen 4.3.23

Oshyan

We are not able to reproduce this behavior nor have we had any other reports of it that I'm aware of. So we can't really diagnose it.

Last we heard you were working on some comparatively low-end hardware so I'm inclined to think it may be related to that in some way. Please don't go into the minimum system requirements discussion again. We are working to provide better guidelines on that, but it should be understood that "minimum" means it is the minimum to run the software, not to have a great experience. Wikipedia puts it this way when talking about "recommended system requirements" vs "minimum system requirements":

"Generally speaking, [recommended system requirements] is a better guideline than minimum system requirements in order to have a fully usable and enjoyable experience with a software."

We do not yet have a "recommended system requirements", but we hope to create one.

- Oshyan

WAS

#2
Quote from: Oshyan on July 17, 2019, 03:51:18 PM
We are not able to reproduce this behavior nor have we had any other reports of it that I'm aware of. So we can't really diagnose it.

Last we heard you were working on some comparatively low-end hardware so I'm inclined to think it may be related to that in some way. Please don't go into the minimum system requirements discussion again. We are working to provide better guidelines on that, but it should be understood that "minimum" means it is the minimum to run the software, not to have a great experience. Wikipedia puts it this way when talking about "recommended system requirements" vs "minimum system requirements":

"Generally speaking, [recommended system requirements] is a better guideline than minimum system requirements in order to have a fully usable and enjoyable experience with a software."

We do not yet have a "recommended system requirements", but we hope to create one.

- Oshyan

I'd be happy to send you guys the file. It renders significantly faster at horizon level with sky in the shot than looking down at the object. Perhaps due to stray lines of sight for reflections (coming from the sky, bouncing to the sky).

It'd probably be best though not to instigate discussion (it's a bad habit as a staff member), especially linking to Wikipedia as you often do, which is most certainly not a definitive source. Most developers actually target minimum to recommended specs. Including drivers updates, etc for better performance. They don't just go "hmm, this seems reasonable enough" (without actual testing) and leave it at that. Lol They go through  alpha, and beta testing phases on full ranges of consumer hardware to make these conclusions -- based on functionality reports of users (it works) or play-ability (it runs at 30fps at least).

And it's not like I can't go ahead and use any other renderer with substantially faster results, ray traced, path traced, or otherwise. I just ran some octane demos on unity with real-time path tracing better quality than PT MPD 0.5 AA4 in real time lmao.

And I'm pretty sure i have seen people note different render times on PT. The chart you released vs hardware indicates a pretty wide range of variance.

And what is comparatively "low-end" mean, in general in these cases? Again, your new workstations you routinely upgrade? Not the best way to develop at all. No hardware standardization to know how it'll run scaled. Lol

The "performance" in TG gets muddled really fast when you're just throwing more cores at it to off-set actual work calculations.

In general as far as this issue goes, changing viewport angle doesn't really equate to CPU when one scenario renders almost 10,000x faster. Lol Immediately seems like something to do with calculations, maybe render settings going crazy, etc.

Oshyan

Yes, please send us the file to test.

You're implying you have some inside knowledge about how we chose the minimum system specs, how we do testing, etc. Where do you get that knowledge I wonder? We're not Autodesk, with access to a full lab of computer hardware to test on, but we have a pretty clear idea of how Terragen performs across a range of systems.

We also absolutely *do* pay attention to user feedback. We have lots of people using older machines and yet nobody else is reporting some of the issues you do. Those that do run into some issues seem to reasonably accept that using older hardware is going to result in a compromised experience.

As far as testing other renderers, I'm not going to make any claim that Terragen is as good or efficient as a dedicated renderer like Octane. However unless your tests on any other renderer are with a similar scene, that includes dynamic/runtime displacement on a large (terrain-wide) scale for example, then the comparison is fairly inapplicable and doesn't tell us much about the source of the problem. Using a non-comparable test to imply some kind of root issue in Terragen's renderer isn't really useful.

Keep in mind I'm not saying there is no problem in the path tracer. I'm simply saying we haven't been able to replicate the issues like this that you are reporting. And unless we are able to do that, we can't determine the cause and fix any potential problems. So please do send us your file.

- Oshyan

WAS

Quote from: Oshyan on July 17, 2019, 04:25:06 PM
You're implying you have some inside knowledge about how we chose the minimum system specs, how we do testing, etc. Where do you get that knowledge I wonder? We're not Autodesk, with access to a full lab of computer hardware to test on, but we have a pretty clear idea of how Terragen performs across a range of systems.

Your public discussions on how alpha testing works, your publicly released benchmark chart for PT, and how you choose to benchmark based on TGD scenarios (static, non-dynamic).

Quote from: Oshyan on July 17, 2019, 04:25:06 PMAs far as testing other renderers, I'm not going to make any claim that Terragen is as good or efficient as a dedicated renderer like Octane. However unless your tests on any other renderer are with a similar scene, that includes dynamic/runtime displacement on a large (terrain-wide) scale for example, then the comparison is fairly inapplicable and doesn't tell us much about the source of the problem. Using a non-comparable test to imply some kind of root issue in Terragen's renderer isn't really useful.

No, a real-time scene won't give you a comparable displacement result. But my scene has no displacement but low level 0.002-0.0001 range scale on sphere objects. That shouldn't really be a issue here. In fact probably far more computation in the real-time city scapes with all that reflection (which is what my scene pretty much just is).

But other systems, do use run-time displacement. Often a displaced plane for a ground shot is used (like terragens planet) when not working with baked heightmaps for a large landscape.

Quote from: Oshyan on July 17, 2019, 04:25:06 PM
Keep in mind I'm not saying there is no problem in the path tracer. I'm simply saying we haven't been able to replicate the issues like this that you are reporting. And unless we are able to do that, we can't determine the cause and fix any potential problems. So please do send us your file.

- Oshyan

I would immediately surmise that half the issues I have where render time is off by an hour wouldn't be noticeable scaled to your systems specs. So you you may not even notice the minutes/seconds offset to take note and dive deeper.

Y

WAS

#5
Sent two files to the support email.  Feel free to let me know how those textures work too lol I've only managed a 800x400 and the surfaces just blend too much so small lol

Oshyan

Files received but are identical. Additionally, is path tracing supposed to be enabled? Because it's not. Maybe you sent the wrong files?

P.S. The benchmark we released was not intended to be used for path tracing. That's why its setup to not use path tracing.

- Oshyan

WAS

Quote from: Oshyan on July 17, 2019, 04:55:22 PM
Files received but are identical. Additionally, is path tracing supposed to be enabled? Because it's not. Maybe you sent the wrong files?

P.S. The benchmark we released was not intended to be used for path tracing. That's why its setup to not use path tracing.

- Oshyan

Looks like I missed the copy view button.

Feel free to let me know how those textures work too lol I've only managed a 800x400 and the surfaces just blend. Can't squint far enough. Lol

I would say though that planetside alpha/beta testing is focusing on waaay to high-end of hardware to really catch oddities in performance for a conclusive min-recommended spec. Even recommended shouldn't be based in absolute overkill of the latest hardware and CPU releases.

No offense here at all, but that's what bad game developers do, they develop on high-end machines they constantly replace, and their game is absolute crap on most actual consumer gaming systems, PUBG and Ring of Elysium to name a couple are hated for this.

archonforest

I think the minimum System CPU for TG is a quad i7 around and above 3Ghz. Preferably a newer generation. (Or an equivalent AMD). I actually tested TG on an i3-not latest, i5-not latest and now on a newer i7-6700. An i3 and an i5 can run TG of course but it is not productive at all. The earlier mentioned i7-6700 is kind of okay. You can get things done with it in a reasonable time.

I think PS should not invest any time to test and test and test...with lower class CPUs than an i7 or similar AMD.
Dell T5500 with Dual Hexa Xeon CPU 3Ghz, 32Gb ram, GTX 1080
Amiga 1200 8Mb ram, 8Gb ssd

WAS

#9
Quote from: archonforest on July 18, 2019, 06:40:11 AM
I think the minimum System CPU for TG is a quad i7 around and above 3Ghz. Preferably a newer generation. (Or an equivalent AMD). I actually tested TG on an i3-not latest, i5-not latest and now on a newer i7-6700. An i3 and an i5 can run TG of course but it is not productive at all. The earlier mentioned i7-6700 is kind of okay. You can get things done with it in a reasonable time.

I think PS should not invest any time to test and test and test...with lower class CPUs than an i7 or similar AMD.

I was very productive until last year on a x5450. Again, this software has not changed since 2009. It's inherently a lightweight program. The only quantative variables is actual rendering and RAM for scenes/pops. Which can most certainly see improvements as TG isn't even as fast as other CPU renderers. You can literally decompile this version and TG2 and see it's mainly be built ontop of with new shaders and and little rewrote. All these things do not impact performance of the renderer, or shouldn't, that would indicate the whole framework is falling apart and needs replacement.

Even the changes to the main renderer have improved upon what was available in 2009 in the performance area lol

Again, again, again, again. This is rendering software which is entirely scalable to even running on XP under my phone at a snail's pace. The only thing that is required is time. However, dramatic changes in computation time for simple changes in camera view for PT immediately suggests something is wrong performance wise. Not my CPU. If it was you are admitting the software isn't optimized and somehow has issues with certain CPUs and definitely should have attention as it no way can be asserted it is only MY CPU, but rather a instruction set/architecture/brand issue.

Masking problems with performance is negligent. Take a look at other renderer software, they target a CPU figure a CPU combination that can get the jobs done. Clearly going higher will only provide higher perfromance, but none of these suites focus on workstation tier CPUs lol Inherently it's rendering, and scalable (or there is a problem). Maya, Blender, C4D, Octane, etc, etc, etc, the list goes on. I've seen none that target workstation CPUs, regardless of their CPU rendering. Nor do none of them just tell people to get a workstation tier CPU in support...

I'm not saying TG should work fast on my system, but a clear radical change in render time by 8 hours is insane and hints at some weird oddity, which I'm trying to figure out... We see tray render times a lot, whether by mistake from render settings or something unknown.

Man, imagine working with SGI and complaining that their systems can't handle it cause how long it takes to just process changes let alone rendering (hence render farms...). xD

Oshyan

I have tested your scene with and without PT. While the camera-up version (that shows some sky) takes a bit longer to render, it is nowhere near the level that you reported. In non-PT it is 32m56s vs. 33m45s. In PT it's 5h36m vs. 7h59m. I am not certain of the reason for the difference but I would guess that the lower angle simply creates more interreflection that needs to be rendered, and that's the most intensive aspect of this scene by far. It also increases draw distance, showing the background mountains, as well as making direct atmosphere volumetric calculations necessary due to the visibility of the sky. All of these are understandable contributing factors to a longer render time.

I also found that both scenes use 100% of the CPU for the majority of the render, only reducing slowly toward the end as the few remaining buckets are rendered by individual threads which are taking longer. This happens in your scene particularly around areas of interreflection which, as I mentioned, are the most intensive in this setup.

This is clearly a very demanding and render time intensive scene with PT enabled. Matt has suggested that using the built-in displaceable spheres might be one factor in it being slower than it could be. So you might consider using an imported sphere instead. An upcoming update will have a built-in non-displaceable sphere that should calculate faster for situations where you don't need the displacement effect.

I will also note that memory use can be fairly high on this scene, depending on the number of threads. On my system with 32 threads it used about 20GB when calculating scene 2, with the camera showing some of the clouds and horizon. I don't know what machine you're working on at this point, I recall you had only 4GB of RAM for a while, but hopefully you have a bit more than that now. Still, 20GB is quite a lot. The memory use will of course be less with fewer threads.

As you can see this was tested on a high-end machine. But unless virtual memory or some other massive performance killer is involved, render time scales pretty linearly. You can clearly see this on our benchmark result lists. So obviously these renders would take a lot longer on your hardware, but the important thing I was testing here is whether the difference between the two camera perspectives creates an *unreasonable* difference in render time, and I have demonstrated that it does not, and hypothesized a plausible explanation for the difference that does exist.

At any rate since I'm still unable to reproduce your reported problems I can only conclude again that some local, system-specific issue is occurring, such as VM paging to disk.

- Oshyan

WAS

Quote from: Oshyan on July 18, 2019, 05:41:56 PM
I have tested your scene with and without PT. While the camera-up version (that shows some sky) takes a bit longer to render, it is nowhere near the level that you reported. In non-PT it is 32m56s vs. 33m45s. In PT it's 5h36m vs. 7h59m. I am not certain of the reason for the difference but I would guess that the lower angle simply creates more interreflection that needs to be rendered, and that's the most intensive aspect of this scene by far. It also increases draw distance, showing the background mountains, as well as making direct atmosphere volumetric calculations necessary due to the visibility of the sky. All of these are understandable contributing factors to a longer render time.

I also found that both scenes use 100% of the CPU for the majority of the render, only reducing slowly toward the end as the few remaining buckets are rendered by individual threads which are taking longer. This happens in your scene particularly around areas of interreflection which, as I mentioned, are the most intensive in this setup.

This is clearly a very demanding and render time intensive scene with PT enabled. Matt has suggested that using the built-in displaceable spheres might be one factor in it being slower than it could be. So you might consider using an imported sphere instead. An upcoming update will have a built-in non-displaceable sphere that should calculate faster for situations where you don't need the displacement effect.

I will also note that memory use can be fairly high on this scene, depending on the number of threads. On my system with 32 threads it used about 20GB when calculating scene 2, with the camera showing some of the clouds and horizon. I don't know what machine you're working on at this point, I recall you had only 4GB of RAM for a while, but hopefully you have a bit more than that now. Still, 20GB is quite a lot. The memory use will of course be less with fewer threads.

As you can see this was tested on a high-end machine. But unless virtual memory or some other massive performance killer is involved, render time scales pretty linearly. You can clearly see this on our benchmark result lists. So obviously these renders would take a lot longer on your hardware, but the important thing I was testing here is whether the difference between the two camera perspectives creates an *unreasonable* difference in render time, and I have demonstrated that it does not, and hypothesized a plausible explanation for the difference that does exist.

At any rate since I'm still unable to reproduce your reported problems I can only conclude again that some local, system-specific issue is occurring, such as VM paging to disk.

- Oshyan

See, the issue I was having was reversed. The horizion view rendered a full row of buckets in 18 minutes working on second line of 4 buckets, while the top down render where I figured I didn't need the sky in the shot randomly was at 8 hours, with only 1 row of buckets done and 4-6 buckets on the second row. Something happened that caused the scene to just crawl.

I can never get steady 100% usage with PT on this AMD CPU. It' usually 80-90% where the standard render is 100%+ (and fans go crazy; literally not even CPU stress tests stress a CPU as bad as TG, that needs looking into too. Usually only a issue with v3 clouds).

I haven't had 4GB of ram for probably 5-6 years. I've used 8GB for awhile plus 32GB commit. Now I use 16GB and 16GB commit. This shouldn't be a factor because if RAM was a issue an illegal exception would occur and guaranteed TG wouldn't handle it and it would CTD.

Matt

WAS, what was RAM use like when this happened? Maybe you were hitting VM hard and it was paging to disk?

There are all sorts of reasons why hardware requirements can change depending on the camera position. If your scene does a lot of ray tracing between displaceable objects (e.g. interreflections), I am dynamically creating micropolygons in the subdiv cache to support this. So RAM use will vary. If you hit VM hard then it will slow down massively, and the solution might be to get more RAM. Does that mean the minimum requirements are wrong? I think it just means they won't be enough for everything you try to do, just like you wouldn't expect a renderer to be able to load bigger models than you have RAM for and wouldn't blame it on the minimum requirements. There are all sorts of scenes that weren't practical to render in 2009 but are now with better hardware.
Just because milk is white doesn't mean that clouds are made of milk.

WAS

Quote from: Matt on July 18, 2019, 05:54:18 PM
WAS, what was RAM use like when this happened? Maybe you were hitting VM hard and it was paging to disk?

There are all sorts of reasons why hardware requirements can change depending on the camera position. If your scene does a lot of ray tracing between displaceable objects (e.g. interreflections), I am dynamically creating micropolygons in the subdiv cache to support this. So RAM use will vary. If you hit VM hard then it will slow down massively, and the solution might be to get more RAM. Does that mean the minimum requirements are wrong? I think it just means they won't be enough for everything you try to do, just like you wouldn't expect a renderer to be able to load bigger models than you have RAM for and wouldn't blame it on the minimum requirements. There are all sorts of scenes that weren't practical to render in 2009 but are now with better hardware.

I was well below maximums (I *believe* around 10gb physical). TG seems to scale this sort of work depending on the RAM you have so long as it's not physical objects taking up RAM.

Oshyan

Then the issue being reversed is probably down to those particular buckets taking a long time on the interreflections, as I mentioned. Did you let either render finish? The render time distribution may be uneven, but that doesn't mean that total render time will be so dramatically uneven.

Also can you lay out your basic hardware spec here just so we know? I'm sure you've posted it before, but I don't recall, and I'll write it down this time.

- Oshyan