Terragen 4 crashes at end of render when rendering 360 sky

Started by jlipstock, July 03, 2019, 06:58:55 PM

Previous topic - Next topic

jlipstock

Hey guys,

I have the most recent update, and when I try to render a high sampled, 360, and 16k sky image it crashes at the end. It renders fine until the very end when it's doing it's final calculations and it can't handle it. If I lower the samples it'll go through, but I don't want to lower the samples and shouldn't have to. Does it at least save a temp version somewhere? Any suggestions? There's no error message, it just closes the software completely.

Thanks

WAS

#1
Sounds like a memory issue.

Could you post some system information? 16k is going to hit a pretty good junk of memory. When dealing with raw 16k images myself, I can easily use up more than 50% of my memory, and with layers/effects creating duplications in memory can create limitations. For example, photoshop will start dumping to scratch and taking up severe disk space (up to 20gb for a 16k image; and that's on top of full memory at 16gb) with multiple layers / effects / masks

Oshyan

I would agree the first thing to look at would be memory issues. I had planned to tackle the email version of this request shortly, but maybe I'll dig into it here.

- Oshyan

Oshyan

OK, responding to the support request here (in the future you can just post here, we'll answer it, or email, either one we'll respond to directly).

First, just quickly, as far as rendering something "in the cloud", Pixel Plow can handle most high resolution images by splitting it up into tiles across multiple machines. There are some non-tile-compatible settings in Terragen, mostly the post effects like bloom and starburst, so you'll want to be aware of that. But otherwise it can be a good solution.

Now as to why it's crashing, we definitely need to know more about your machine's hardware to get a good idea of what might be happening. But as WAS said, a 16k image is going to require a fair amount of memory, especially as AA is increased.

The fact that it hangs at the end might indicate an issue with the "GISD" pass, if that's enabled. In fact on such a high resolution GISD can take quite a while to complete, so if you haven't left it for an hour (I know, sounds like a long time), then I would wonder whether it's just been working on something and not giving feedback in the UI, which can happen on really high resolution renders.

I think the best things to do would be to monitor both memory and CPU use toward the end of the render. If the memory use is near your total system memory, that's going to cause problems. And if it seems to hang but the CPU is still being used, that might indicate it's doing work on GISD and just not updating the UI, which means you might want to try to wait it out, give it at least 30 minutes at that resolution.

As for whether the image is saved, there is an optional automatic render output to file, but it only saves once a render completes. So if it's really crashing before it finishes, nothing would have been saved unfortunately.

Hopefully you can provide some more info and we can better help you diagnose this.

- Oshyan

jlipstock

I have an i7-6700K CPU 4ghz with 32GB of memory. On a windows 10.

I don't have any passes enabled. I'm just rendering the sky. I also, don't forcefully close it. It crashes on its own. I'm assuming it has to do with my memory. I've never had this issue before with other renderers though...

It takes 3 days to render one frame on my computer. is there a calculator to figure out how much faster it would be if I had a faster cpu? I'm trying to figure out how much it would be for one frame using pixel plow.


Oshyan

Pixel Plow has their own calculator, though it is approximate: http://www.pixelplow.net/pricing/

To find out how much faster a new CPU would be, the ideal thing to do is run our handy benchmark and compare with the other CPUs on the list!
https://planetside.co.uk/terragen-4-benchmark/

While fast at the time, the i7-6700k is a bit old now. 3 days may be necessary given the high render resolution, but optimization is also possible in many cases. If you send us your TGD we can usually help with that.

- Oshyan

WAS

#6
Honestly I would say that this is related to a memory limitation, while 32gb seems like a lot, you've got a cloudscape in 360 going on, a lot of raw data is being held on to. I'm still not exactly sure how things are handled when the post-processing pass happens before the render is finalized, but from my understanding of the effects applied it may multiply what's already in memory to do GI shading. Additionally you AA is going to add too all this as well.

I wonder if you could hang around the machine when it start finalizing and get a good readout of your memory usage. Mind you it's possible TG cannot allocate all your memory, some of it may be allocated elsewhere, or what TG wants your RAM can't supply so it's possible to see a crash due to memory at even 80-90% usage (or lower depending on what's going on with background tasks).

jlipstock

here's my TGD file if you want to take a look at it.

jlipstock

I did your benchmark and I got 13:46

My render settings for my clouds are probably too high. I cranked it up because I wanted to make sure there was no noise, since I kept getting it. But still, it shouldn't crash when I crank up the settings...

DannyG

Its a memory issue, right out of the gate it is consuming 22GB... let that go for a few hours at that resolution and you will tap with 32GB. Defer atm with v3 or easy clouds is heavy on your system. I personally would not use defer. Just do some crop tests with increased samples in the cloud & atm layers until you fine the sweet spot. That would be my advise, others may chime in. Good luck bro
New World Digital Art
NwdaGroup.com
Media: facebook|Twitter|Instagram

WAS

Quote from: DannyG on July 04, 2019, 08:20:45 PM
Its a memory issue, right out of the gate it is consuming 22GB... let that go for a few hours at that resolution and you will tap with 32GB. Defer atm with v3 or easy clouds is heavy on your system. I personally would not use defer. Just do some crop tests with increased samples in the cloud & atm layers until you fine the sweet spot. That would be my advise, others may chime in. Good luck bro

Echo what Danny said about cloud settings. Perhaps my cloud settings share here may help: https://planetside.co.uk/forums/index.php/topic,26647.msg265743.html#msg265743

Oshyan

Good error catching is important, but many applications still crash when they run out of memory. It's difficult to avoid that with current architectures, as far as I've seen. And when you increase settings, you use more system resources, that is essentially inevitable. So at some point as you increase settings you'll run into memory limits and potential crashes.

As Danny said, after taking a look at your file it's fairly obvious to me why you're having problems. When I *start* to render the memory use shoots up to 25+GB. That's just during the GI cache phase, before primary rendering even begins. So it's not surprising to me that it taxes your system at some point with 32GB of RAM.

First and most importantly, rendering at high resolution simply does take a lot of memory. Terragen renders at 32bits per channel (RGBA), which requires over 2GB of memory *just* for the base image in memory at that resolution. Then more memory is needed for render buffers, antialiasing, voxel cache (for v3 clouds), GI cache, etc. Almost all of those use more memory at higher resolution.

So there may not actually be a lot you can do to significantly reduce memory use here when rendering at this resolution and antialiasing level. You can for example try rendering a totally empty scene at the same resolution and AA you'll see it's already using a fair amount of memory even without the clouds (22GB on my system). And thus no amount of cloud optimization is likely to truly fix this given your RAM limits.

But since resolution alone is one of the biggest factors, one thing that *might* have a notable impact is to either render from the command line and use the -cropout option to only render the crop area (this will output an image with vertical resolution much less than what you've specified because the bottom part of the image is cropped out). I'm not certain but I *believe* -cropout should use less memory. Alternatively, remove the crop and adjust your render resolution and possibly FoV to get the same view without the crop, i.e. roughly half as many vertical pixels as you have now (not certain this is possible with spherical due to distortion if the camera angle shifts).

Since this scene seems to be just near your limits of memory at the moment, the above might just resolve this particular situation. Something else to try, since it crashes right near the end apparently, is to disable GI Surface Details (Image Pass Tab). This *only* affects surfaces, but with it enabled, it's probably wasting memory and processor time for no benefit.

I'll try to make a follow-up post about optimizing render settings for faster render times at equivalent or higher quality. But avoiding the crash seemed like the higher priority for the moment.

Btw Danny, Defer Atmo doesn't really use more memory to render for equivalent quality levels. At least not in my tests. It does sometimes take longer to render vs. without, but the situations where that's true are fewer and fewer, especially with v3 clouds, and as the renderer has been optimized further since Defer Atmo's introduction.

- Oshyan

jlipstock

Thanks for the suggestions guys! Is making the crop through the command line different from doing it through the UI? I've been cropping it that way. I haven't been rendering the whole thing. Does that mean it still calculating the rest of the image?

Oshyan

It does not "calculate" the rest of the image, but the total size of the image buffers is equal to the total render resolution, *not* to the crop region. In other words the number of pixels that are saved to your resulting image is what dictates the in-memory image size. The crop just controls what area is calculated, not what area is *stored*, if that makes sense. The unrendered (not calculated) area is just stored as black, but in an uncompressed format (which in-memory data is) black still takes up space. ;)

- Oshyan

WAS

Quote from: Oshyan on July 05, 2019, 12:19:12 AM
The unrendered (not calculated) area is just stored as black, but in an uncompressed format (which in-memory data is) black still takes up space. ;)

- Oshyan

? Significantly less space, though. A blank 16k tiff/png is only about a MB compared to a texture map/displacement or whatever at 250-500mb+...

What's that like for Terragen's raw output?