Path Tracing Slowdown

Started by WAS, July 17, 2019, 12:56:23 PM

Previous topic - Next topic

Matt

Quote from: WASasquatch on July 18, 2019, 05:53:48 PM
I haven't had 4GB of ram for probably 5-6 years. I've used 8GB for awhile plus 32GB commit. Now I use 16GB and 16GB commit. This shouldn't be a factor because if RAM was a issue an illegal exception would occur and guaranteed TG wouldn't handle it and it would CTD.

How can you guarantee that? I can't guarantee that and I wrote the thing! Running out of RAM might just crash render threads and/or cause all sorts of silent problems.

If your environment is set up to put a hard limit on memory, as it sounds like you've done, then I wonder if that's what's causing these issues.
Just because milk is white doesn't mean that clouds are made of milk.

WAS

Quote from: Oshyan on July 18, 2019, 05:56:33 PM
Then the issue being reversed is probably down to those particular buckets taking a long time on the interreflections, as I mentioned. Did you let either render finish? The render time distribution may be uneven, but that doesn't mean that total render time will be so dramatically uneven.

Also can you lay out your basic hardware spec here just so we know? I'm sure you've posted it before, but I don't recall, and I'll write it down this time.

- Oshyan

Yeah, let me install speccy and just export a sheet.

Also you are probably right about reflections. This has happened to me before with ocean scenes + reflective objects (a boat) where when I moved from my working camera to my render camera the render times didn't coincide at all.

Also I'm curious why when TG does hit RAM maximums, we do get just a illegal exception in windows logs rather than it being handled? For example, other software I'll get a windows prompt that I'm out of memory. I've never seen that with TG in all the years I've used it. It just crashes.

WAS

#17
Quote from: Matt on July 18, 2019, 05:59:32 PM
Quote from: WASasquatch on July 18, 2019, 05:53:48 PM
I haven't had 4GB of ram for probably 5-6 years. I've used 8GB for awhile plus 32GB commit. Now I use 16GB and 16GB commit. This shouldn't be a factor because if RAM was a issue an illegal exception would occur and guaranteed TG wouldn't handle it and it would CTD.

How can you guarantee that? I can't guarantee that and I wrote the thing! Running out of RAM might just crash render threads and/or cause all sorts of silent problems.

If your environment is set up to put a hard limit on memory, as it sounds like you've done, then I wonder if that's what's causing these issues.

See post above. I've never seen a out of memory prompt from windows allowing me to close things and save Terragen. Even at school with 32gb on TG2. Never. If you run out of memory, terragen just closes to the desktop, and leaves a error in windows log.

Heck even with technical support you both go immediately to the 3d preview, or ram, when TG CTD.

Oshyan

Separately, to clarify, "minimum specs" is exactly intended to set a baseline - i.e. a minimum - for what will run the software and do basic operations. In the game world it means you can run a game in low resolution and low detail and a low but playable frame rate. Since game content is more finite and specific (the developers know what assets are in the game, how much geometry is in a given part of a given level, etc.), and there is a minimum frame rate for human perception of interactivity (around 30fps, obviously higher is better!), it's fairly easy for game publishers to say what the minimum hardware should be.

Open-ended creativity applications are much harder to specify min spec for because it depends dramatically on what you are doing with it. A simple scene with a few single objects or small populations and basic shaders is going to be very different from a complex scene with multiple levels of displacement, intersecting v3 cloud layers, large populations, etc. Sometimes it's not necessarily intuitive what will be demanding in Terragen, not because it's not logical, simply because it's hard to intuit what takes a lot of calculation without knowing the internals of how Terragen works. We are working to make that kind of info more clear and available.

In the case of this scene, each of these balls uses a shader network with more than 30 (and in one case more than 40) nodes. They're beautiful materials, but that's a lot of nodes to evaluate for the path tracer, I think. Many similar scenes in other apps would not be using any procedurals at all - or very few - mostly relying on image maps. In the case of image-based materials the lookup is quite simple, just checking values in a texture map, rather than calculating a bunch of math to figure out pixel color for 30+ nodes in sequence.

- Oshyan

Oshyan

The "out of memory" messages come from Windows, not Terragen. We don't do anything special to manage memory (especially virtual memory), Windows is supposed to handle that stuff. And in the case of VM it usually does so silently. In fact I was just testing something else (1 billion instance population) and I was running Performance Monitor, so I could see it paging to disk, but I got no warning messages of any kind.

I'm honestly not sure what specific situation causes the "low on memory" messages, but I have gotten them when using Terragen on very rare occasions. I suspect it's when both physical *and* virtual memory is completely full.

- Oshyan

WAS

#20
Quote from: Oshyan on July 18, 2019, 06:04:23 PM
Separately, to clarify, "minimum specs" is exactly intended to set a baseline - i.e. a minimum - for what will run the software and do basic operations.

No, it doesn't, Oshyan. Stop it. You won't refute [hundreds of] thousands of applications. It means it will ALL run, not basic operations. Lol It just may not be as fast. Same quantifies to gaming as well, slower, thus you COULD do a lower resolution and detail.

Even with software if something won't function because of a limitation of hardware it's noted right there in the requirements such as GPU limitations of instruction sets for certain features. Whether it's 64-bit, whether you need a certain cores such as recommended vs optimal on something like Maya for 4 cores vs 8. But they aren't going to say a CPU than just takes longer is the cause of technical issues. Lol it's supported, thus they look at it constructively and don't attack a users hardware.

Quote from: Oshyan on July 18, 2019, 06:06:50 PM
The "out of memory" messages come from Windows, not Terragen. We don't do anything special to manage memory (especially virtual memory), Windows is supposed to handle that stuff. And in the case of VM it usually does so silently. In fact I was just testing something else (1 billion instance population) and I was running Performance Monitor, so I could see it paging to disk, but I got no warning messages of any kind.

I'm honestly not sure what specific situation causes the "low on memory" messages, but I have gotten them when using Terragen on very rare occasions. I suspect it's when both physical *and* virtual memory is completely full.

- Oshyan

Stop. Oshyan. Stop. Windows is only going to have an illegal exception to memory access because of the software, and thus, couldn't prompt the user that they are out of memory...

An error occurs, and terragen cannot recover and Windows kills the process to save the computer from possible corruption or crashes....

Matt

Quote from: WASasquatch on July 18, 2019, 05:56:09 PM
Quote from: Matt on July 18, 2019, 05:54:18 PM
WAS, what was RAM use like when this happened? Maybe you were hitting VM hard and it was paging to disk?
I was well below maximums (I *believe* around 10gb physical). TG seems to scale this sort of work depending on the RAM you have so long as it's not physical objects taking up RAM.

Ah that's good to know. I guess the most likely explanation is just the sheer bulk of calculations going on with all those interreflections running in one or two threads. It might work out better by overriding the bucket size to something smaller. Usually something between 32 and 64 is a good range.
Just because milk is white doesn't mean that clouds are made of milk.

Oshyan

I wasn't talking about the crashes (it was not crashing in your case, so it's a separate topic). I was talking about the out of memory *messages*, which you brought up. Those are, as I said, generated by Windows and not the application.

The crashes are absolutely an issue, and Terragen could probably take a more active role in memory management, and especially memory-related error catching. But that's not the issue here as far as I can see.

- Oshyan

WAS

Quote from: Oshyan on July 18, 2019, 06:20:15 PM
I wasn't talking about the crashes (it was not crashing in your case, so it's a separate topic). I was talking about the out of memory *messages*, which you brought up. Those are, as I said, generated by Windows and not the application.

The crashes are absolutely an issue, and Terragen could probably take a more active role in memory management, and especially memory-related error catching. But that's not the issue here as far as I can see.

- Oshyan

I know this. The point of mentioning that is hinting at the instability of the software. Windows can't report this if an error occurs during illegal memory access and it kills it.

Here is my system specs trimmed of all the nonsense speccy throws in (holy crap it puts EVERYTHING in there).

I immediately notice something interesting. I am fairly certain the memory is 1333mhz, and here I get a report of 666mhz. Yuck.

Oshyan

Thanks, I'll add it to our support case reference here so we'll have it in the future.

- Oshyan

WAS

#25
Also, it may be a good idea to add some memory management to TG. It can dynamically monitor memory and know (well guess; especially when it comes to loading assets) if it can't do something, similar to other programs that have their own memory prompts killing their internal processes.

Pretty sure you can even call a ram-dump before opening new projects -- which may catch leaked memory.

Oshyan

Agreed that better memory issue handling would be good to add to Terragen. I see that I did misunderstand what you originally wrote above about the *application* throwing a Windows error message about memory. So I apologize for misconstruing what you wrote. I know there are other apps that handle memory better, but I'm curious of what specific examples you have in mind.

- Oshyan

WAS

#27
Quote from: Oshyan on July 18, 2019, 06:43:23 PM
Agreed that better memory issue handling would be good to add to Terragen. I see that I did misunderstand what you originally wrote above about the *application* throwing a Windows error message about memory. So I apologize for misconstruing what you wrote. I know there are other apps that handle memory better, but I'm curious of what specific examples you have in mind.

- Oshyan

One example I can think of off the top of my head is Photoshop, with all it's extensive third party plugins it can catch if the plugin has become unresponsive cause of ram usage or whatever. Some programs (more in the past) would eat up memory using their libraries of high definition TIFFs or PNGs at huge resolutions to add all those silly stylized effects that were so popular for a long time. Some of them were good uses like one that was used for lens flares.

So back when you only had 2gb of RAM cause it was 2003, a plugin could just destroy it and the scratch you have set lol

Err, I'll add that i've seen this cause weeeeird issues though. Like lets say a plugin is open, and it disabled parts of photoshop UI cause it would break the plugin, while, photoshop just killed the plugin and yelled at you, but failed to unlock most of the PS GUI. Lol

WAS

Could even be this sort of handling of processes came about because of liabilities. Someone downloading some crap plugin that killed their computer and than blaming Adobe.