Planetside Software Forums

General => Terragen Discussion => Topic started by: WAS on July 17, 2019, 12:56:23 PM

Title: Path Tracing Slowdown
Post by: WAS on July 17, 2019, 12:56:23 PM
Still having issues with the path tracer sometimes using no more then 20% of the CPU.

Last time I was going to render my aluminum balls. And so I started, it, whatever, it finished 1 row and 4 buckets in 18 minutes while I went to the bathroom before bed. When I came back I was like "Wow, it's going to probably take longer with the sky"... So all I did was raise the camera, and tilt it down at the balls so only the ground and balls were in the shot. I clicked render and went to bed.

In the morning, there was only one row and 6 buckets done... in 8 hours and 45 minutes... since it was 18 minutes for slightly less, that's like what? a 10000x increase in render time for nothing but a camera adjustment?

What gives? Nothing changed in the scene but camera angle. And again, the CPU wasn't doing anything really (it was literally at 24% when I stopped the render) and all fans were off and computer cooled like it has been like this all night. What could be doing this? This seems to happen to me all the time when I Path Trace where I just waste an entire night to nothing.

I mentioned this with the Benchmark where I got significantly longer render times on a second render.  I'm on Terragen 4.3.23
Title: Re: Path Tracing Slowdown
Post by: Oshyan on July 17, 2019, 03:51:18 PM
We are not able to reproduce this behavior nor have we had any other reports of it that I'm aware of. So we can't really diagnose it.

Last we heard you were working on some comparatively low-end hardware so I'm inclined to think it may be related to that in some way. Please don't go into the minimum system requirements discussion again. We are working to provide better guidelines on that, but it should be understood that "minimum" means it is the minimum to run the software, not to have a great experience. Wikipedia puts it this way when talking about "recommended system requirements (https://en.wikipedia.org/wiki/System_requirements#Recommended_system_requirements)" vs "minimum system requirements":

"Generally speaking, [recommended system requirements] is a better guideline than minimum system requirements in order to have a fully usable and enjoyable experience with a software."

We do not yet have a "recommended system requirements", but we hope to create one.

- Oshyan
Title: Re: Path Tracing Slowdown
Post by: WAS on July 17, 2019, 04:15:56 PM
Quote from: Oshyan on July 17, 2019, 03:51:18 PM
We are not able to reproduce this behavior nor have we had any other reports of it that I'm aware of. So we can't really diagnose it.

Last we heard you were working on some comparatively low-end hardware so I'm inclined to think it may be related to that in some way. Please don't go into the minimum system requirements discussion again. We are working to provide better guidelines on that, but it should be understood that "minimum" means it is the minimum to run the software, not to have a great experience. Wikipedia puts it this way when talking about "recommended system requirements (https://en.wikipedia.org/wiki/System_requirements#Recommended_system_requirements)" vs "minimum system requirements":

"Generally speaking, [recommended system requirements] is a better guideline than minimum system requirements in order to have a fully usable and enjoyable experience with a software."

We do not yet have a "recommended system requirements", but we hope to create one.

- Oshyan

I'd be happy to send you guys the file. It renders significantly faster at horizon level with sky in the shot than looking down at the object. Perhaps due to stray lines of sight for reflections (coming from the sky, bouncing to the sky).

It'd probably be best though not to instigate discussion (it's a bad habit as a staff member), especially linking to Wikipedia as you often do, which is most certainly not a definitive source. Most developers actually target minimum to recommended specs. Including drivers updates, etc for better performance. They don't just go "hmm, this seems reasonable enough" (without actual testing) and leave it at that. Lol They go through  alpha, and beta testing phases on full ranges of consumer hardware to make these conclusions -- based on functionality reports of users (it works) or play-ability (it runs at 30fps at least).

And it's not like I can't go ahead and use any other renderer with substantially faster results, ray traced, path traced, or otherwise. I just ran some octane demos on unity with real-time path tracing better quality than PT MPD 0.5 AA4 in real time lmao.

And I'm pretty sure i have seen people note different render times on PT. The chart you released vs hardware indicates a pretty wide range of variance.

And what is comparatively "low-end" mean, in general in these cases? Again, your new workstations you routinely upgrade? Not the best way to develop at all. No hardware standardization to know how it'll run scaled. Lol

The "performance" in TG gets muddled really fast when you're just throwing more cores at it to off-set actual work calculations.

In general as far as this issue goes, changing viewport angle doesn't really equate to CPU when one scenario renders almost 10,000x faster. Lol Immediately seems like something to do with calculations, maybe render settings going crazy, etc.
Title: Re: Path Tracing Slowdown
Post by: Oshyan on July 17, 2019, 04:25:06 PM
Yes, please send us the file to test.

You're implying you have some inside knowledge about how we chose the minimum system specs, how we do testing, etc. Where do you get that knowledge I wonder? We're not Autodesk, with access to a full lab of computer hardware to test on, but we have a pretty clear idea of how Terragen performs across a range of systems.

We also absolutely *do* pay attention to user feedback. We have lots of people using older machines and yet nobody else is reporting some of the issues you do. Those that do run into some issues seem to reasonably accept that using older hardware is going to result in a compromised experience.

As far as testing other renderers, I'm not going to make any claim that Terragen is as good or efficient as a dedicated renderer like Octane. However unless your tests on any other renderer are with a similar scene, that includes dynamic/runtime displacement on a large (terrain-wide) scale for example, then the comparison is fairly inapplicable and doesn't tell us much about the source of the problem. Using a non-comparable test to imply some kind of root issue in Terragen's renderer isn't really useful.

Keep in mind I'm not saying there is no problem in the path tracer. I'm simply saying we haven't been able to replicate the issues like this that you are reporting. And unless we are able to do that, we can't determine the cause and fix any potential problems. So please do send us your file.

- Oshyan
Title: Re: Path Tracing Slowdown
Post by: WAS on July 17, 2019, 04:32:12 PM
Quote from: Oshyan on July 17, 2019, 04:25:06 PM
You're implying you have some inside knowledge about how we chose the minimum system specs, how we do testing, etc. Where do you get that knowledge I wonder? We're not Autodesk, with access to a full lab of computer hardware to test on, but we have a pretty clear idea of how Terragen performs across a range of systems.

Your public discussions on how alpha testing works, your publicly released benchmark chart for PT, and how you choose to benchmark based on TGD scenarios (static, non-dynamic).

Quote from: Oshyan on July 17, 2019, 04:25:06 PMAs far as testing other renderers, I'm not going to make any claim that Terragen is as good or efficient as a dedicated renderer like Octane. However unless your tests on any other renderer are with a similar scene, that includes dynamic/runtime displacement on a large (terrain-wide) scale for example, then the comparison is fairly inapplicable and doesn't tell us much about the source of the problem. Using a non-comparable test to imply some kind of root issue in Terragen's renderer isn't really useful.

No, a real-time scene won't give you a comparable displacement result. But my scene has no displacement but low level 0.002-0.0001 range scale on sphere objects. That shouldn't really be a issue here. In fact probably far more computation in the real-time city scapes with all that reflection (which is what my scene pretty much just is).

But other systems, do use run-time displacement. Often a displaced plane for a ground shot is used (like terragens planet) when not working with baked heightmaps for a large landscape.

Quote from: Oshyan on July 17, 2019, 04:25:06 PM
Keep in mind I'm not saying there is no problem in the path tracer. I'm simply saying we haven't been able to replicate the issues like this that you are reporting. And unless we are able to do that, we can't determine the cause and fix any potential problems. So please do send us your file.

- Oshyan

I would immediately surmise that half the issues I have where render time is off by an hour wouldn't be noticeable scaled to your systems specs. So you you may not even notice the minutes/seconds offset to take note and dive deeper.

Y
Title: Re: Path Tracing Slowdown
Post by: WAS on July 17, 2019, 04:43:03 PM
Sent two files to the support email.  Feel free to let me know how those textures work too lol I've only managed a 800x400 and the surfaces just blend too much so small lol
Title: Re: Path Tracing Slowdown
Post by: Oshyan on July 17, 2019, 04:55:22 PM
Files received but are identical. Additionally, is path tracing supposed to be enabled? Because it's not. Maybe you sent the wrong files?

P.S. The benchmark we released was not intended to be used for path tracing. That's why its setup to not use path tracing.

- Oshyan
Title: Re: Path Tracing Slowdown
Post by: WAS on July 17, 2019, 04:59:07 PM
Quote from: Oshyan on July 17, 2019, 04:55:22 PM
Files received but are identical. Additionally, is path tracing supposed to be enabled? Because it's not. Maybe you sent the wrong files?

P.S. The benchmark we released was not intended to be used for path tracing. That's why its setup to not use path tracing.

- Oshyan

Looks like I missed the copy view button.

Feel free to let me know how those textures work too lol I've only managed a 800x400 and the surfaces just blend. Can't squint far enough. Lol

I would say though that planetside alpha/beta testing is focusing on waaay to high-end of hardware to really catch oddities in performance for a conclusive min-recommended spec. Even recommended shouldn't be based in absolute overkill of the latest hardware and CPU releases.

No offense here at all, but that's what bad game developers do, they develop on high-end machines they constantly replace, and their game is absolute crap on most actual consumer gaming systems, PUBG and Ring of Elysium to name a couple are hated for this.
Title: Re: Path Tracing Slowdown
Post by: archonforest on July 18, 2019, 06:40:11 AM
I think the minimum System CPU for TG is a quad i7 around and above 3Ghz. Preferably a newer generation. (Or an equivalent AMD). I actually tested TG on an i3-not latest, i5-not latest and now on a newer i7-6700. An i3 and an i5 can run TG of course but it is not productive at all. The earlier mentioned i7-6700 is kind of okay. You can get things done with it in a reasonable time.

I think PS should not invest any time to test and test and test...with lower class CPUs than an i7 or similar AMD.
Title: Re: Path Tracing Slowdown
Post by: WAS on July 18, 2019, 02:08:51 PM
Quote from: archonforest on July 18, 2019, 06:40:11 AM
I think the minimum System CPU for TG is a quad i7 around and above 3Ghz. Preferably a newer generation. (Or an equivalent AMD). I actually tested TG on an i3-not latest, i5-not latest and now on a newer i7-6700. An i3 and an i5 can run TG of course but it is not productive at all. The earlier mentioned i7-6700 is kind of okay. You can get things done with it in a reasonable time.

I think PS should not invest any time to test and test and test...with lower class CPUs than an i7 or similar AMD.

I was very productive until last year on a x5450. Again, this software has not changed since 2009. It's inherently a lightweight program. The only quantative variables is actual rendering and RAM for scenes/pops. Which can most certainly see improvements as TG isn't even as fast as other CPU renderers. You can literally decompile this version and TG2 and see it's mainly be built ontop of with new shaders and and little rewrote. All these things do not impact performance of the renderer, or shouldn't, that would indicate the whole framework is falling apart and needs replacement.

Even the changes to the main renderer have improved upon what was available in 2009 in the performance area lol

Again, again, again, again. This is rendering software which is entirely scalable to even running on XP under my phone at a snail's pace. The only thing that is required is time. However, dramatic changes in computation time for simple changes in camera view for PT immediately suggests something is wrong performance wise. Not my CPU. If it was you are admitting the software isn't optimized and somehow has issues with certain CPUs and definitely should have attention as it no way can be asserted it is only MY CPU, but rather a instruction set/architecture/brand issue.

Masking problems with performance is negligent. Take a look at other renderer software, they target a CPU figure a CPU combination that can get the jobs done. Clearly going higher will only provide higher perfromance, but none of these suites focus on workstation tier CPUs lol Inherently it's rendering, and scalable (or there is a problem). Maya, Blender, C4D, Octane, etc, etc, etc, the list goes on. I've seen none that target workstation CPUs, regardless of their CPU rendering. Nor do none of them just tell people to get a workstation tier CPU in support...

I'm not saying TG should work fast on my system, but a clear radical change in render time by 8 hours is insane and hints at some weird oddity, which I'm trying to figure out... We see tray render times a lot, whether by mistake from render settings or something unknown.

Man, imagine working with SGI and complaining that their systems can't handle it cause how long it takes to just process changes let alone rendering (hence render farms...). xD
Title: Re: Path Tracing Slowdown
Post by: Oshyan on July 18, 2019, 05:41:56 PM
I have tested your scene with and without PT. While the camera-up version (that shows some sky) takes a bit longer to render, it is nowhere near the level that you reported. In non-PT it is 32m56s vs. 33m45s. In PT it's 5h36m vs. 7h59m. I am not certain of the reason for the difference but I would guess that the lower angle simply creates more interreflection that needs to be rendered, and that's the most intensive aspect of this scene by far. It also increases draw distance, showing the background mountains, as well as making direct atmosphere volumetric calculations necessary due to the visibility of the sky. All of these are understandable contributing factors to a longer render time.

I also found that both scenes use 100% of the CPU for the majority of the render, only reducing slowly toward the end as the few remaining buckets are rendered by individual threads which are taking longer. This happens in your scene particularly around areas of interreflection which, as I mentioned, are the most intensive in this setup.

This is clearly a very demanding and render time intensive scene with PT enabled. Matt has suggested that using the built-in displaceable spheres might be one factor in it being slower than it could be. So you might consider using an imported sphere instead. An upcoming update will have a built-in non-displaceable sphere that should calculate faster for situations where you don't need the displacement effect.

I will also note that memory use can be fairly high on this scene, depending on the number of threads. On my system with 32 threads it used about 20GB when calculating scene 2, with the camera showing some of the clouds and horizon. I don't know what machine you're working on at this point, I recall you had only 4GB of RAM for a while, but hopefully you have a bit more than that now. Still, 20GB is quite a lot. The memory use will of course be less with fewer threads.

As you can see this was tested on a high-end machine. But unless virtual memory or some other massive performance killer is involved, render time scales pretty linearly. You can clearly see this on our benchmark result lists. So obviously these renders would take a lot longer on your hardware, but the important thing I was testing here is whether the difference between the two camera perspectives creates an *unreasonable* difference in render time, and I have demonstrated that it does not, and hypothesized a plausible explanation for the difference that does exist.

At any rate since I'm still unable to reproduce your reported problems I can only conclude again that some local, system-specific issue is occurring, such as VM paging to disk.

- Oshyan
Title: Re: Path Tracing Slowdown
Post by: WAS on July 18, 2019, 05:53:48 PM
Quote from: Oshyan on July 18, 2019, 05:41:56 PM
I have tested your scene with and without PT. While the camera-up version (that shows some sky) takes a bit longer to render, it is nowhere near the level that you reported. In non-PT it is 32m56s vs. 33m45s. In PT it's 5h36m vs. 7h59m. I am not certain of the reason for the difference but I would guess that the lower angle simply creates more interreflection that needs to be rendered, and that's the most intensive aspect of this scene by far. It also increases draw distance, showing the background mountains, as well as making direct atmosphere volumetric calculations necessary due to the visibility of the sky. All of these are understandable contributing factors to a longer render time.

I also found that both scenes use 100% of the CPU for the majority of the render, only reducing slowly toward the end as the few remaining buckets are rendered by individual threads which are taking longer. This happens in your scene particularly around areas of interreflection which, as I mentioned, are the most intensive in this setup.

This is clearly a very demanding and render time intensive scene with PT enabled. Matt has suggested that using the built-in displaceable spheres might be one factor in it being slower than it could be. So you might consider using an imported sphere instead. An upcoming update will have a built-in non-displaceable sphere that should calculate faster for situations where you don't need the displacement effect.

I will also note that memory use can be fairly high on this scene, depending on the number of threads. On my system with 32 threads it used about 20GB when calculating scene 2, with the camera showing some of the clouds and horizon. I don't know what machine you're working on at this point, I recall you had only 4GB of RAM for a while, but hopefully you have a bit more than that now. Still, 20GB is quite a lot. The memory use will of course be less with fewer threads.

As you can see this was tested on a high-end machine. But unless virtual memory or some other massive performance killer is involved, render time scales pretty linearly. You can clearly see this on our benchmark result lists. So obviously these renders would take a lot longer on your hardware, but the important thing I was testing here is whether the difference between the two camera perspectives creates an *unreasonable* difference in render time, and I have demonstrated that it does not, and hypothesized a plausible explanation for the difference that does exist.

At any rate since I'm still unable to reproduce your reported problems I can only conclude again that some local, system-specific issue is occurring, such as VM paging to disk.

- Oshyan

See, the issue I was having was reversed. The horizion view rendered a full row of buckets in 18 minutes working on second line of 4 buckets, while the top down render where I figured I didn't need the sky in the shot randomly was at 8 hours, with only 1 row of buckets done and 4-6 buckets on the second row. Something happened that caused the scene to just crawl.

I can never get steady 100% usage with PT on this AMD CPU. It' usually 80-90% where the standard render is 100%+ (and fans go crazy; literally not even CPU stress tests stress a CPU as bad as TG, that needs looking into too. Usually only a issue with v3 clouds).

I haven't had 4GB of ram for probably 5-6 years. I've used 8GB for awhile plus 32GB commit. Now I use 16GB and 16GB commit. This shouldn't be a factor because if RAM was a issue an illegal exception would occur and guaranteed TG wouldn't handle it and it would CTD.
Title: Re: Path Tracing Slowdown
Post by: Matt on July 18, 2019, 05:54:18 PM
WAS, what was RAM use like when this happened? Maybe you were hitting VM hard and it was paging to disk?

There are all sorts of reasons why hardware requirements can change depending on the camera position. If your scene does a lot of ray tracing between displaceable objects (e.g. interreflections), I am dynamically creating micropolygons in the subdiv cache to support this. So RAM use will vary. If you hit VM hard then it will slow down massively, and the solution might be to get more RAM. Does that mean the minimum requirements are wrong? I think it just means they won't be enough for everything you try to do, just like you wouldn't expect a renderer to be able to load bigger models than you have RAM for and wouldn't blame it on the minimum requirements. There are all sorts of scenes that weren't practical to render in 2009 but are now with better hardware.
Title: Re: Path Tracing Slowdown
Post by: WAS on July 18, 2019, 05:56:09 PM
Quote from: Matt on July 18, 2019, 05:54:18 PM
WAS, what was RAM use like when this happened? Maybe you were hitting VM hard and it was paging to disk?

There are all sorts of reasons why hardware requirements can change depending on the camera position. If your scene does a lot of ray tracing between displaceable objects (e.g. interreflections), I am dynamically creating micropolygons in the subdiv cache to support this. So RAM use will vary. If you hit VM hard then it will slow down massively, and the solution might be to get more RAM. Does that mean the minimum requirements are wrong? I think it just means they won't be enough for everything you try to do, just like you wouldn't expect a renderer to be able to load bigger models than you have RAM for and wouldn't blame it on the minimum requirements. There are all sorts of scenes that weren't practical to render in 2009 but are now with better hardware.

I was well below maximums (I *believe* around 10gb physical). TG seems to scale this sort of work depending on the RAM you have so long as it's not physical objects taking up RAM.
Title: Re: Path Tracing Slowdown
Post by: Oshyan on July 18, 2019, 05:56:33 PM
Then the issue being reversed is probably down to those particular buckets taking a long time on the interreflections, as I mentioned. Did you let either render finish? The render time distribution may be uneven, but that doesn't mean that total render time will be so dramatically uneven.

Also can you lay out your basic hardware spec here just so we know? I'm sure you've posted it before, but I don't recall, and I'll write it down this time.

- Oshyan
Title: Re: Path Tracing Slowdown
Post by: Matt on July 18, 2019, 05:59:32 PM
Quote from: WASasquatch on July 18, 2019, 05:53:48 PM
I haven't had 4GB of ram for probably 5-6 years. I've used 8GB for awhile plus 32GB commit. Now I use 16GB and 16GB commit. This shouldn't be a factor because if RAM was a issue an illegal exception would occur and guaranteed TG wouldn't handle it and it would CTD.

How can you guarantee that? I can't guarantee that and I wrote the thing! Running out of RAM might just crash render threads and/or cause all sorts of silent problems.

If your environment is set up to put a hard limit on memory, as it sounds like you've done, then I wonder if that's what's causing these issues.
Title: Re: Path Tracing Slowdown
Post by: WAS on July 18, 2019, 06:00:57 PM
Quote from: Oshyan on July 18, 2019, 05:56:33 PM
Then the issue being reversed is probably down to those particular buckets taking a long time on the interreflections, as I mentioned. Did you let either render finish? The render time distribution may be uneven, but that doesn't mean that total render time will be so dramatically uneven.

Also can you lay out your basic hardware spec here just so we know? I'm sure you've posted it before, but I don't recall, and I'll write it down this time.

- Oshyan

Yeah, let me install speccy and just export a sheet.

Also you are probably right about reflections. This has happened to me before with ocean scenes + reflective objects (a boat) where when I moved from my working camera to my render camera the render times didn't coincide at all.

Also I'm curious why when TG does hit RAM maximums, we do get just a illegal exception in windows logs rather than it being handled? For example, other software I'll get a windows prompt that I'm out of memory. I've never seen that with TG in all the years I've used it. It just crashes.
Title: Re: Path Tracing Slowdown
Post by: WAS on July 18, 2019, 06:01:54 PM
Quote from: Matt on July 18, 2019, 05:59:32 PM
Quote from: WASasquatch on July 18, 2019, 05:53:48 PM
I haven't had 4GB of ram for probably 5-6 years. I've used 8GB for awhile plus 32GB commit. Now I use 16GB and 16GB commit. This shouldn't be a factor because if RAM was a issue an illegal exception would occur and guaranteed TG wouldn't handle it and it would CTD.

How can you guarantee that? I can't guarantee that and I wrote the thing! Running out of RAM might just crash render threads and/or cause all sorts of silent problems.

If your environment is set up to put a hard limit on memory, as it sounds like you've done, then I wonder if that's what's causing these issues.

See post above. I've never seen a out of memory prompt from windows allowing me to close things and save Terragen. Even at school with 32gb on TG2. Never. If you run out of memory, terragen just closes to the desktop, and leaves a error in windows log.

Heck even with technical support you both go immediately to the 3d preview, or ram, when TG CTD.
Title: Re: Path Tracing Slowdown
Post by: Oshyan on July 18, 2019, 06:04:23 PM
Separately, to clarify, "minimum specs" is exactly intended to set a baseline - i.e. a minimum - for what will run the software and do basic operations. In the game world it means you can run a game in low resolution and low detail and a low but playable frame rate. Since game content is more finite and specific (the developers know what assets are in the game, how much geometry is in a given part of a given level, etc.), and there is a minimum frame rate for human perception of interactivity (around 30fps, obviously higher is better!), it's fairly easy for game publishers to say what the minimum hardware should be.

Open-ended creativity applications are much harder to specify min spec for because it depends dramatically on what you are doing with it. A simple scene with a few single objects or small populations and basic shaders is going to be very different from a complex scene with multiple levels of displacement, intersecting v3 cloud layers, large populations, etc. Sometimes it's not necessarily intuitive what will be demanding in Terragen, not because it's not logical, simply because it's hard to intuit what takes a lot of calculation without knowing the internals of how Terragen works. We are working to make that kind of info more clear and available.

In the case of this scene, each of these balls uses a shader network with more than 30 (and in one case more than 40) nodes. They're beautiful materials, but that's a lot of nodes to evaluate for the path tracer, I think. Many similar scenes in other apps would not be using any procedurals at all - or very few - mostly relying on image maps. In the case of image-based materials the lookup is quite simple, just checking values in a texture map, rather than calculating a bunch of math to figure out pixel color for 30+ nodes in sequence.

- Oshyan
Title: Re: Path Tracing Slowdown
Post by: Oshyan on July 18, 2019, 06:06:50 PM
The "out of memory" messages come from Windows, not Terragen. We don't do anything special to manage memory (especially virtual memory), Windows is supposed to handle that stuff. And in the case of VM it usually does so silently. In fact I was just testing something else (1 billion instance population) and I was running Performance Monitor, so I could see it paging to disk, but I got no warning messages of any kind.

I'm honestly not sure what specific situation causes the "low on memory" messages, but I have gotten them when using Terragen on very rare occasions. I suspect it's when both physical *and* virtual memory is completely full.

- Oshyan
Title: Re: Path Tracing Slowdown
Post by: WAS on July 18, 2019, 06:10:59 PM
Quote from: Oshyan on July 18, 2019, 06:04:23 PM
Separately, to clarify, "minimum specs" is exactly intended to set a baseline - i.e. a minimum - for what will run the software and do basic operations.

No, it doesn't, Oshyan. Stop it. You won't refute [hundreds of] thousands of applications. It means it will ALL run, not basic operations. Lol It just may not be as fast. Same quantifies to gaming as well, slower, thus you COULD do a lower resolution and detail.

Even with software if something won't function because of a limitation of hardware it's noted right there in the requirements such as GPU limitations of instruction sets for certain features. Whether it's 64-bit, whether you need a certain cores such as recommended vs optimal on something like Maya for 4 cores vs 8. But they aren't going to say a CPU than just takes longer is the cause of technical issues. Lol it's supported, thus they look at it constructively and don't attack a users hardware.

Quote from: Oshyan on July 18, 2019, 06:06:50 PM
The "out of memory" messages come from Windows, not Terragen. We don't do anything special to manage memory (especially virtual memory), Windows is supposed to handle that stuff. And in the case of VM it usually does so silently. In fact I was just testing something else (1 billion instance population) and I was running Performance Monitor, so I could see it paging to disk, but I got no warning messages of any kind.

I'm honestly not sure what specific situation causes the "low on memory" messages, but I have gotten them when using Terragen on very rare occasions. I suspect it's when both physical *and* virtual memory is completely full.

- Oshyan

Stop. Oshyan. Stop. Windows is only going to have an illegal exception to memory access because of the software, and thus, couldn't prompt the user that they are out of memory...

An error occurs, and terragen cannot recover and Windows kills the process to save the computer from possible corruption or crashes....
Title: Re: Path Tracing Slowdown
Post by: Matt on July 18, 2019, 06:12:53 PM
Quote from: WASasquatch on July 18, 2019, 05:56:09 PM
Quote from: Matt on July 18, 2019, 05:54:18 PM
WAS, what was RAM use like when this happened? Maybe you were hitting VM hard and it was paging to disk?
I was well below maximums (I *believe* around 10gb physical). TG seems to scale this sort of work depending on the RAM you have so long as it's not physical objects taking up RAM.

Ah that's good to know. I guess the most likely explanation is just the sheer bulk of calculations going on with all those interreflections running in one or two threads. It might work out better by overriding the bucket size to something smaller. Usually something between 32 and 64 is a good range.
Title: Re: Path Tracing Slowdown
Post by: Oshyan on July 18, 2019, 06:20:15 PM
I wasn't talking about the crashes (it was not crashing in your case, so it's a separate topic). I was talking about the out of memory *messages*, which you brought up. Those are, as I said, generated by Windows and not the application.

The crashes are absolutely an issue, and Terragen could probably take a more active role in memory management, and especially memory-related error catching. But that's not the issue here as far as I can see.

- Oshyan
Title: Re: Path Tracing Slowdown
Post by: WAS on July 18, 2019, 06:22:19 PM
Quote from: Oshyan on July 18, 2019, 06:20:15 PM
I wasn't talking about the crashes (it was not crashing in your case, so it's a separate topic). I was talking about the out of memory *messages*, which you brought up. Those are, as I said, generated by Windows and not the application.

The crashes are absolutely an issue, and Terragen could probably take a more active role in memory management, and especially memory-related error catching. But that's not the issue here as far as I can see.

- Oshyan

I know this. The point of mentioning that is hinting at the instability of the software. Windows can't report this if an error occurs during illegal memory access and it kills it.

Here is my system specs trimmed of all the nonsense speccy throws in (holy crap it puts EVERYTHING in there).

I immediately notice something interesting. I am fairly certain the memory is 1333mhz, and here I get a report of 666mhz. Yuck.
Title: Re: Path Tracing Slowdown
Post by: Oshyan on July 18, 2019, 06:24:03 PM
Thanks, I'll add it to our support case reference here so we'll have it in the future.

- Oshyan
Title: Re: Path Tracing Slowdown
Post by: WAS on July 18, 2019, 06:26:43 PM
Also, it may be a good idea to add some memory management to TG. It can dynamically monitor memory and know (well guess; especially when it comes to loading assets) if it can't do something, similar to other programs that have their own memory prompts killing their internal processes.

Pretty sure you can even call a ram-dump before opening new projects -- which may catch leaked memory.
Title: Re: Path Tracing Slowdown
Post by: Oshyan on July 18, 2019, 06:43:23 PM
Agreed that better memory issue handling would be good to add to Terragen. I see that I did misunderstand what you originally wrote above about the *application* throwing a Windows error message about memory. So I apologize for misconstruing what you wrote. I know there are other apps that handle memory better, but I'm curious of what specific examples you have in mind.

- Oshyan
Title: Re: Path Tracing Slowdown
Post by: WAS on July 18, 2019, 06:48:45 PM
Quote from: Oshyan on July 18, 2019, 06:43:23 PM
Agreed that better memory issue handling would be good to add to Terragen. I see that I did misunderstand what you originally wrote above about the *application* throwing a Windows error message about memory. So I apologize for misconstruing what you wrote. I know there are other apps that handle memory better, but I'm curious of what specific examples you have in mind.

- Oshyan

One example I can think of off the top of my head is Photoshop, with all it's extensive third party plugins it can catch if the plugin has become unresponsive cause of ram usage or whatever. Some programs (more in the past) would eat up memory using their libraries of high definition TIFFs or PNGs at huge resolutions to add all those silly stylized effects that were so popular for a long time. Some of them were good uses like one that was used for lens flares.

So back when you only had 2gb of RAM cause it was 2003, a plugin could just destroy it and the scratch you have set lol

Err, I'll add that i've seen this cause weeeeird issues though. Like lets say a plugin is open, and it disabled parts of photoshop UI cause it would break the plugin, while, photoshop just killed the plugin and yelled at you, but failed to unlock most of the PS GUI. Lol
Title: Re: Path Tracing Slowdown
Post by: WAS on July 18, 2019, 06:52:50 PM
Could even be this sort of handling of processes came about because of liabilities. Someone downloading some crap plugin that killed their computer and than blaming Adobe.