TG3 Crashes

Started by jaf, December 24, 2014, 02:56:16 PM

Previous topic - Next topic

jaf

I've been getting intermittent crashes lately and finally feel I've narrowed it down enough to post some details here (my system is in my signature.)  I've been able to rule out memory and heat problems but I'm 99% sure it caused by NOT having the preview paused and initiating a render.  I typically us min 1 & max 64 threads and the crash occurs during the populating phase.  If I pause the preview, I don't get the crash.  So this is not a big deal to me since it's easy enough to avoid.  So if you experience crashes, it's one thing that's easy to try.

I wonder if TG should have a "lockout feature" whenever a render or population update is active?

(04Dec20) Ryzen 1800x, 970 EVO 1TB M.2 SSD, Corsair Vengeance 64GB DDR4 3200 Mem,  EVGA GeForce GTX 1080 Ti FTW3 Graphics 457.51 (04Dec20), Win 10 Pro x64, Terragen Pro 4.5.43 Frontier, BenchMark 0:10:02

Kadri


Sometimes it happens to me too. I paused it too for that.
But lately by using the more basic preview option (by not pressing the "renderers" button above in the menu bar) i got the same result (no crashes).

WAS

I've been having random crashing changing values within shaders when they're in the high negatives or simply low values. I'll change a xxxxx.02 to xxxxx.03 and it'll lock up. Sometimes it doesn't though.

I have gotten the population crashes too, though mine seem to be mainly related to generating meshes on Rocks, as there is no  actual progress bar, and just a blank box that pops up and closes (which would contain a progress bar if it was some sort of insane mesh). I am assuming all these boxes are causing issues with WDM and causing PIDs to go haywire and the app crash. Maybe those windows should be somehow tied into a overall progress like the actual population and not for each individual mesh when you have some 100+ variations.

All in all the problems for me aren't many. I just cranked down variations to 30, and with shaders, I save often. Lol

fleetfoot

I just started using T3 Creative on a 64bit Windows 7 i7 PC with 12 gigs of ram and an nVidia GeForce GTX 650 graphics card w/2 gigs GDDR5.
I'm getting crashes of the application fairly often for no apparent reason. A couple of times it froze the whole PC and required a reboot.
One thing I can repeat often is a crash when trying to link an image with a tgd using the library. It crashes more often than not but surprisingly, sometimes it works.
I'm a newbie and I don't think I'm stressing the system or the software much at all as I try to learn it. My perception so far is that the application is fairly unstable at this point.
I also have a 2011 iMac with moderate capabilities which I will install T3 on to see if it displays similar behavior.
Regards,
fleetfoot

Oshyan

#4
Thanks for the reports folks, and sorry for the instability you're experiencing.

Regarding the population-related problems, we have become aware of a possible intermittent crash in that code and we're looking into it, but it's tough to pin down so far. If you can narrow it down to specific actions, etc. that's very helpful. The possibility that it may be related to "not having the 3D preview paused" is interesting and makes me wonder if it means the 3D preview is always in an updating process (i.e. still rendering) when you start the full render. Can anyone clarify that or provide any further details?

Fleetfoot, your experience does not sound normal, that's the good news. So hopefully we can resolve it. Most of the time frequent stability issues come down either to hardware issues, i.e. memory, CPU heat, etc. (most people think it's not their hardware, of course, but it still sometimes is :D ) OR graphics card driver problems.

To my recollection we have not heard of any instances of Terragen taking down the entire system that were *not* related to underlying hardware or driver issues, and this makes sense because modern versions of Windows (Vista and newer, but especially Win7 or 8 ) are really quite robust and applications basically aren't allowed to do things that fundamentally break the OS, except in rare circumstances. So the fact that a hard crash occurred does indicate to me that something outside of TG is not working quite right. It's worth doing a test of your memory (Memtest86+ if possible) and a basic burn-in for an hour or two (e.g. Prime95) while monitoring temperature of the CPU. I realize that this may sound ridiculous as it's quite likely no other apps are causing problems, but from long experience it seems that this is one of the best places to start looking when things like this come up.

Now, it could be graphics related; that can cause some pretty hard crashes if the drivers aren't setup to handle problematic situations gracefully. These issues are quite frustrating as we're not really using any terribly demanding or complex OpenGL functions in Terragen, it just seems to be that the graphics card manufacturers are more focused on game performance and DirectX than stability, OpenGL, and 3D graphics baselines. Sometimes updating the graphics card drivers helps, but unfortunately sometimes *older* drivers actually work better. It's worth trying an update to start though. You can also try closing the 3D preview for a while as a test and see if it appears to be more stable (you can even start TG with the -no3dpreview option to disable it by default).

Hopefully that helps. Let us know if you find out anything further. We'll certainly do our best to make Terragen stable on your machine!

- Oshyan

jaf

I haven't had a crash since I have paused the preview render before doing a population update or starting a render (that does a population update.) 

I believe there was a relatively recent update to the preview window making it faster (multi-threading?)  Or was it populating, or both?  Anyway, my system shows 98 to 99% cpu usage when populating and I'm not sure if the preview is locked out -- have to look and see.  Also, I've been using population preview colors a lot.
(04Dec20) Ryzen 1800x, 970 EVO 1TB M.2 SSD, Corsair Vengeance 64GB DDR4 3200 Mem,  EVGA GeForce GTX 1080 Ti FTW3 Graphics 457.51 (04Dec20), Win 10 Pro x64, Terragen Pro 4.5.43 Frontier, BenchMark 0:10:02

WAS

#6
Quote from: Oshyan on December 25, 2014, 01:33:56 AM
Thanks for the reports folks, and sorry for the instability you're experiencing.

Regarding the population-related problems, we have become aware of a possible intermittent crash in that code and we're looking into it, but it's tough to pin down so far. If you can narrow it down to specific actions, etc. that's very helpful. The possibility that it may be related to "not having the 3D preview paused" is interesting and makes me wonder if it means the 3D preview is always in an updating process (i.e. still rendering) when you start the full render. Can anyone clarify that or provide any further details?

Fleetfoot, your experience does not sound normal, that's the good news. So hopefully we can resolve it. Most of the time frequent stability issues come down either to hardware issues, i.e. memory, CPU heat, etc. (most people think it's not their hardware, of course, but it still sometimes is :D ) OR graphics card driver problems.

To my recollection we have not heard of any instances of Terragen taking down the entire system that were *not* related to underlying hardware or driver issues, and this makes sense because modern versions of Windows (Vista and newer, but especially Win7 or 8 ) are really quite robust and applications basically aren't allowed to do things that fundamentally break the OS, except in rare circumstances. So the fact that a hard crash occurred does indicate to me that something outside of TG is not working quite right. It's worth doing a test of your memory (Memtest86+ if possible) and a basic burn-in for an hour or two (e.g. Prime95) while monitoring temperature of the CPU. I realize that this may sound ridiculous as it's quite likely no other apps are causing problems, but from long experience it seems that this is one of the best places to start looking when things like this come up.

Now, it could be graphics related; that can cause some pretty hard crashes if the drivers aren't setup to handle problematic situations gracefully. These issues are quite frustrating as we're not really using any terribly demanding or complex OpenGL functions in Terragen, it just seems to be that the graphics card manufacturers are more focused on game performance and DirectX than stability, OpenGL, and 3D graphics baselines. Sometimes updating the graphics card drivers helps, but unfortunately sometimes *older* drivers actually work better. It's worth trying an update to start though. You can also try closing the 3D preview for a while as a test and see if it appears to be more stable (you can even start TG with the -no3dpreview option to disable it by default).

Hopefully that helps. Let us know if you find out anything further. We'll certainly do our best to make Terragen stable on your machine!

- Oshyan

Terragen does render the preview while a full render is going. For example, if I don't pause, and start a render, when I come back and minimize the render, or finish it the preview will be "Finished rendering" when it wasn't even at 5% when starting the render.

I also seemed to notice last night Terragen was restarting it's render preview when dragging nodes around. This I can not totally confirm as I'm waiting on my PSU and having fun on a Rasberry Pi, so this could be CPU L2 Cache related, but if I remember correctly the preview was continuously going back to 5% detail every time I dragged a node while rearranging, which caused the app to lock up during each drag/click.

Oshyan

Interesting Sasquatch, you've found at least 1 bug! Indeed, if you simply drag a node *while the preview is updating*, it will restart the update process. If you let the preview finish then move a node it doesn't restart. But that first part is definitely a bug, and a rather annoying one too. Well spotted.

Now, the 3D preview definitely should *not* be updating while primary rendering is going on. In my tests just now I was not able to show that it does. If I started a render in the middle of the preview update, the render completed normally, then the restarted and updated until complete. Can you demonstrate different behavior consistently?

- Oshyan

Dune

I've mentioned instability ever since the preview became multithreaded, and made a mental note to save as often as possible. Indeed, even changing a simple number sometimes makes TG freeze/crash, I reported that as well, half a year or more ago. Also dragging the crophandles around starts refreshing the preview, which is quite irritating sometimes. I don't like being negative, but this unpredictable behavior is something that annoyed me all along.

WAS

Quote from: Oshyan on December 30, 2014, 01:55:35 AM
Interesting Sasquatch, you've found at least 1 bug! Indeed, if you simply drag a node *while the preview is updating*, it will restart the update process. If you let the preview finish then move a node it doesn't restart. But that first part is definitely a bug, and a rather annoying one too. Well spotted.

Now, the 3D preview definitely should *not* be updating while primary rendering is going on. In my tests just now I was not able to show that it does. If I started a render in the middle of the preview update, the render completed normally, then the restarted and updated until complete. Can you demonstrate different behavior consistently?

- Oshyan

I think I can confirm that. I do get different behavior. I started some small test renders just to see, and it seems 5/10 times it will not pause the render (timing on clicking "Render Image" while preview is rendering?) The times it did pause the preview, it stayed paused until the render finished, the other 5 times it was rendering both, and actually cause significant slow down with the primary render, and one time; what do you know, one of those weird little bucket cache boxes looking off-place like the cell didn't render correctly.



No settings were changed during the render, that just... happened (?)

jaf

I would really be in favor of a lockout of data entry/modification and preview renders while a population or render is in progress.  Maybe allow file saving, but the rest is all "read only."

Of course this might not be practical from the developers perspective, but it would cut down a lot of new user's frustration (and old user's like me who forget what they are doing.)  :)

I might add the TG3 seems very stable when using other software.  I know it's not recommended, but my job is a 24/7 one and frequently I have to fire up an office program or do some Internet work.  I do try to remember to pause a render, but sometime forget.
(04Dec20) Ryzen 1800x, 970 EVO 1TB M.2 SSD, Corsair Vengeance 64GB DDR4 3200 Mem,  EVGA GeForce GTX 1080 Ti FTW3 Graphics 457.51 (04Dec20), Win 10 Pro x64, Terragen Pro 4.5.43 Frontier, BenchMark 0:10:02

Oshyan

The intended behavior already is that Preview rendering should stop when primary rendering happens. So if that's not the case, as WASasquatch reports, then it's a bug that will be fixed. I should note however that unlike his other bug (moving nodes restarts preview rendering), I haven't been able to reproduce the preview rendering while primary rendering is going on. If anyone else gets this behavior too, please let us know!

The lack of "lock out" for other parameters is more a source of debate at Planetside. Personally I'm very much in agreement that parameter editing should be disallowed, or at least there should be a warning when you attempt it that it's not recommended and may cause unpredictable results. Hopefully that's a change that can be agreed on and made in the future.

- Oshyan

Dune

Well, of course it's rather logical that you shouldn't change parameters when rendering; it's like changing a tyre whilst driving  :P Out of cuiriosity, I attempted it a few times, resulting in crashes or just weird color changes, the crashes sometimes only occurring after the render is starting the bucket with the changes in it (after half an hour or so).

jaf

My real job -- managing a couple commercial buildings -- results in lots of interruptions.  If I had the money I could have a separate computer to do my renders.  SO I'm pausing renders to run a spreadsheet, prepare a memo, scan an invoice, etc.  And I'm getting old and forgetful, so I think my computer should help me out a bit.

What benefit is there in allowing a user to make data changes while a render is in progress?

Okay, here's another "pet peeve."  I make a change in the preview window that requires I click on "Copy this view to the current render camera."  Okay, I remember most of the time, but there's time I don't or am interrupted and forget.  Well, I can just look a the icon, right?  The images below show the difference in the left-most icon.  Maybe a slow "blink" or a more radical change in the icon?

These aren't big deals and I've mentioned before that TG is my favorite application, so my complaints are more questions of "why not..?"  So I can't resist putting in my 2 cents.  :)



(04Dec20) Ryzen 1800x, 970 EVO 1TB M.2 SSD, Corsair Vengeance 64GB DDR4 3200 Mem,  EVGA GeForce GTX 1080 Ti FTW3 Graphics 457.51 (04Dec20), Win 10 Pro x64, Terragen Pro 4.5.43 Frontier, BenchMark 0:10:02

WAS

#14
Quote from: jaf on January 02, 2015, 05:30:17 PM
My real job -- managing a couple commercial buildings -- results in lots of interruptions.  If I had the money I could have a separate computer to do my renders.  SO I'm pausing renders to run a spreadsheet, prepare a memo, scan an invoice, etc.  And I'm getting old and forgetful, so I think my computer should help me out a bit.

What benefit is there in allowing a user to make data changes while a render is in progress?

Okay, here's another "pet peeve."  I make a change in the preview window that requires I click on "Copy this view to the current render camera."  Okay, I remember most of the time, but there's time I don't or am interrupted and forget.  Well, I can just look a the icon, right?  The images below show the difference in the left-most icon.  Maybe a slow "blink" or a more radical change in the icon?

These aren't big deals and I've mentioned before that TG is my favorite application, so my complaints are more questions of "why not..?"  So I can't resist putting in my 2 cents.  :)

I actually was lost when looking at the images. My brain did not allow me to see the change, and this probably stems from the same issue I have in Terragen with cameras. I'm not that old... yet (26) but I have these same "pet peeves"

Also, as far as nodes and rendering.... I don't think the preview should be restarted unless you are plugging/unplugging nodes, as simply moving a node from a group, or changing the syntax of the nodes to look more appealing and organized has no impact on the render. So that is a bit strange it would restart even if I just click and drag a node half a centimeter, causing unnecessary strain on someones computer.

Also, allowing editing of the project while it renders, again, as mentioned above, I don't see any real benefit. Even a slight change in the ground, when rendering the sky, will cause the sky to break when the edits don't effect the sky. And more then 80% of the time, even just renaming a node, or changing a .01 variance will cause it to cash, instead of actually allowing that edit. Again, unnecessary strain on someones computer.

I just lost a PSU from Terragen spiking CPU/RAM usages (specifically forgetting a render was going and changing a PF scale is when it audibly went "POP") because of these issues, and I should have a whole 250 watts to spare. The fact TG can send my computer way past it's hardware limits and programmed restraints is very bad and can kill a computer from something as simple as accidentally changing a node while forgetting a render is underway. Luckily all the seemed to have popped was the PSU, and a single RAM card.