Hey guys,
I have the most recent update, and when I try to render a high sampled, 360, and 16k sky image it crashes at the end. It renders fine until the very end when it's doing it's final calculations and it can't handle it. If I lower the samples it'll go through, but I don't want to lower the samples and shouldn't have to. Does it at least save a temp version somewhere? Any suggestions? There's no error message, it just closes the software completely.
Thanks
Sounds like a memory issue.
Could you post some system information? 16k is going to hit a pretty good junk of memory. When dealing with raw 16k images myself, I can easily use up more than 50% of my memory, and with layers/effects creating duplications in memory can create limitations. For example, photoshop will start dumping to scratch and taking up severe disk space (up to 20gb for a 16k image; and that's on top of full memory at 16gb) with multiple layers / effects / masks
I would agree the first thing to look at would be memory issues. I had planned to tackle the email version of this request shortly, but maybe I'll dig into it here.
- Oshyan
OK, responding to the support request here (in the future you can just post here, we'll answer it, or email, either one we'll respond to directly).
First, just quickly, as far as rendering something "in the cloud", Pixel Plow can handle most high resolution images by splitting it up into tiles across multiple machines. There are some non-tile-compatible settings in Terragen, mostly the post effects like bloom and starburst, so you'll want to be aware of that. But otherwise it can be a good solution.
Now as to why it's crashing, we definitely need to know more about your machine's hardware to get a good idea of what might be happening. But as WAS said, a 16k image is going to require a fair amount of memory, especially as AA is increased.
The fact that it hangs at the end might indicate an issue with the "GISD" pass, if that's enabled. In fact on such a high resolution GISD can take quite a while to complete, so if you haven't left it for an hour (I know, sounds like a long time), then I would wonder whether it's just been working on something and not giving feedback in the UI, which can happen on really high resolution renders.
I think the best things to do would be to monitor both memory and CPU use toward the end of the render. If the memory use is near your total system memory, that's going to cause problems. And if it seems to hang but the CPU is still being used, that might indicate it's doing work on GISD and just not updating the UI, which means you might want to try to wait it out, give it at least 30 minutes at that resolution.
As for whether the image is saved, there is an optional automatic render output to file, but it only saves once a render completes. So if it's really crashing before it finishes, nothing would have been saved unfortunately.
Hopefully you can provide some more info and we can better help you diagnose this.
- Oshyan
I have an i7-6700K CPU 4ghz with 32GB of memory. On a windows 10.
I don't have any passes enabled. I'm just rendering the sky. I also, don't forcefully close it. It crashes on its own. I'm assuming it has to do with my memory. I've never had this issue before with other renderers though...
It takes 3 days to render one frame on my computer. is there a calculator to figure out how much faster it would be if I had a faster cpu? I'm trying to figure out how much it would be for one frame using pixel plow.
Pixel Plow has their own calculator, though it is approximate: http://www.pixelplow.net/pricing/
To find out how much faster a new CPU would be, the ideal thing to do is run our handy benchmark and compare with the other CPUs on the list!
https://planetside.co.uk/terragen-4-benchmark/
While fast at the time, the i7-6700k is a bit old now. 3 days may be necessary given the high render resolution, but optimization is also possible in many cases. If you send us your TGD we can usually help with that.
- Oshyan
Honestly I would say that this is related to a memory limitation, while 32gb seems like a lot, you've got a cloudscape in 360 going on, a lot of raw data is being held on to. I'm still not exactly sure how things are handled when the post-processing pass happens before the render is finalized, but from my understanding of the effects applied it may multiply what's already in memory to do GI shading. Additionally you AA is going to add too all this as well.
I wonder if you could hang around the machine when it start finalizing and get a good readout of your memory usage. Mind you it's possible TG cannot allocate all your memory, some of it may be allocated elsewhere, or what TG wants your RAM can't supply so it's possible to see a crash due to memory at even 80-90% usage (or lower depending on what's going on with background tasks).
here's my TGD file if you want to take a look at it.
I did your benchmark and I got 13:46
My render settings for my clouds are probably too high. I cranked it up because I wanted to make sure there was no noise, since I kept getting it. But still, it shouldn't crash when I crank up the settings...
Its a memory issue, right out of the gate it is consuming 22GB... let that go for a few hours at that resolution and you will tap with 32GB. Defer atm with v3 or easy clouds is heavy on your system. I personally would not use defer. Just do some crop tests with increased samples in the cloud & atm layers until you fine the sweet spot. That would be my advise, others may chime in. Good luck bro
Quote from: DannyG on July 04, 2019, 08:20:45 PM
Its a memory issue, right out of the gate it is consuming 22GB... let that go for a few hours at that resolution and you will tap with 32GB. Defer atm with v3 or easy clouds is heavy on your system. I personally would not use defer. Just do some crop tests with increased samples in the cloud & atm layers until you fine the sweet spot. That would be my advise, others may chime in. Good luck bro
Echo what Danny said about cloud settings. Perhaps my cloud settings share here may help: https://planetside.co.uk/forums/index.php/topic,26647.msg265743.html#msg265743
Good error catching is important, but many applications still crash when they run out of memory. It's difficult to avoid that with current architectures, as far as I've seen. And when you increase settings, you use more system resources, that is essentially inevitable. So at some point as you increase settings you'll run into memory limits and potential crashes.
As Danny said, after taking a look at your file it's fairly obvious to me why you're having problems. When I *start* to render the memory use shoots up to 25+GB. That's just during the GI cache phase, before primary rendering even begins. So it's not surprising to me that it taxes your system at some point with 32GB of RAM.
First and most importantly, rendering at high resolution simply does take a lot of memory. Terragen renders at 32bits per channel (RGBA), which requires over 2GB of memory *just* for the base image in memory at that resolution. Then more memory is needed for render buffers, antialiasing, voxel cache (for v3 clouds), GI cache, etc. Almost all of those use more memory at higher resolution.
So there may not actually be a lot you can do to significantly reduce memory use here when rendering at this resolution and antialiasing level. You can for example try rendering a totally empty scene at the same resolution and AA you'll see it's already using a fair amount of memory even without the clouds (22GB on my system). And thus no amount of cloud optimization is likely to truly fix this given your RAM limits.
But since resolution alone is one of the biggest factors, one thing that *might* have a notable impact is to either render from the command line and use the -cropout option to only render the crop area (this will output an image with vertical resolution much less than what you've specified because the bottom part of the image is cropped out). I'm not certain but I *believe* -cropout should use less memory. Alternatively, remove the crop and adjust your render resolution and possibly FoV to get the same view without the crop, i.e. roughly half as many vertical pixels as you have now (not certain this is possible with spherical due to distortion if the camera angle shifts).
Since this scene seems to be just near your limits of memory at the moment, the above might just resolve this particular situation. Something else to try, since it crashes right near the end apparently, is to disable GI Surface Details (Image Pass Tab). This *only* affects surfaces, but with it enabled, it's probably wasting memory and processor time for no benefit.
I'll try to make a follow-up post about optimizing render settings for faster render times at equivalent or higher quality. But avoiding the crash seemed like the higher priority for the moment.
Btw Danny, Defer Atmo doesn't really use more memory to render for equivalent quality levels. At least not in my tests. It does sometimes take longer to render vs. without, but the situations where that's true are fewer and fewer, especially with v3 clouds, and as the renderer has been optimized further since Defer Atmo's introduction.
- Oshyan
Thanks for the suggestions guys! Is making the crop through the command line different from doing it through the UI? I've been cropping it that way. I haven't been rendering the whole thing. Does that mean it still calculating the rest of the image?
It does not "calculate" the rest of the image, but the total size of the image buffers is equal to the total render resolution, *not* to the crop region. In other words the number of pixels that are saved to your resulting image is what dictates the in-memory image size. The crop just controls what area is calculated, not what area is *stored*, if that makes sense. The unrendered (not calculated) area is just stored as black, but in an uncompressed format (which in-memory data is) black still takes up space. ;)
- Oshyan
Quote from: Oshyan on July 05, 2019, 12:19:12 AM
The unrendered (not calculated) area is just stored as black, but in an uncompressed format (which in-memory data is) black still takes up space. ;)
- Oshyan
? Significantly less space, though. A blank 16k tiff/png is only about a MB compared to a texture map/displacement or whatever at 250-500mb+...
What's that like for Terragen's raw output?
Those are compressed formats (although TIFF has non-compressed modes as well). As far as I'm aware *in memory* image data is not compressed in Terragen. As a good example of this if you load a 2000x2000 pixel JPG image with an Image Map Shader, it could be a mere 1MB on disk, but it will be represented in memory (for Terragen as a renderer) by uncompressed RGB values. 4 million pixels, 24 bits per pixel, 12 million bytes, so about 11.5MB *uncompressed*.
Terragen outputs EXR with compression, and TIFF with optional compression. But again that is when saving to an on-disk format. *In memory* compression is not used. Matt will correct me if I'm wrong. :D
- Oshyan
The -cropoutput command line option uses less memory for cropped renders, so this is worth trying. But it's not just for command line rendering. You can use it with the GUI. If you go to a command line window or console window and run Terragen with -cropoutput somewhere in the command line, you can use the full GUI as normal. Or create a shortcut with this added to the command line options.
Quote from: Matt on July 05, 2019, 05:29:43 AM
The -cropoutput command line option uses less memory for cropped renders, so this is worth trying. But it's not just for command line rendering. You can use it with the GUI. If you go to a command line window or console window and run Terragen with -cropoutput somewhere in the command line, you can use the full GUI as normal. Or create a shortcut with this added to the command line options.
Handy information! I am going to add a shortcut with the command. May come in handy. Thanks for that information.
Quote from: Oshyan on July 04, 2019, 09:18:27 PM
Btw Danny, Defer Atmo doesn't really use more memory to render for equivalent quality levels. At least not in my tests. It does sometimes take longer to render vs. without, but the situations where that's true are fewer and fewer, especially with v3 clouds, and as the renderer has been optimized further since Defer Atmo's introduction.
- Oshyan
Understood, I did not say it did. I said it is heavy on your system. I should have been more clear. Totally agree defer has been optimized big time since the transition from "raytraced". Honestly it is still very heavy on my CPU. If I have heavy clouds in a scene I do not use it at all. The difference in time is est 1/3 less without defer. Yes I tweak samples and crop with defer, it just seems as soon as you enable it your CPU is running up a steep hill. This is a blanket statement of course. Every scene is different.
I would like to add to this discussion with a related point but not exactly the same thing, but it's important to know about it. Sometimes Terragen will appear to be finishing a render, or it might have already finished and POOF it goes waay (crash). When this happens to me it's not a memory issue because I have 128GB and never see my memory going much over 64GB of use.
What I suspect is because I don't always pause the RTP it will start back up after my renders finish, and if RTP runs into a bug somewhere while refining it could cause a crash. I suspect this because I do get my render frames saved, but when I come back to my PC and check Terragen is not running. Of course I get scared, and check to see if my frames were saved and they were. So even if Terragen crashes, always check to see if your frames were saved.
There is also an option in Preferences (File Saving) to save all renders even if you don't chose "Render All to Disk" or "Render sequence" from the Render Panel.
-Derek
Quote from: D.A. Bentley on July 19, 2019, 04:03:47 PM
I would like to add to this discussion with a related point but not exactly the same thing, but it's important to know about it. Sometimes Terragen will appear to be finishing a render, or it might have already finished and POOF it goes waay (crash). When this happens to me it's not a memory issue because I have 128GB and never see my memory going much over 64GB of use.
What I suspect is because I don't always pause the RTP it will start back up after my renders finish, and if RTP runs into a bug somewhere while refining it could cause a crash. I suspect this because I do get my render frames saved, but when I come back to my PC and check Terragen is not running. Of course I get scared, and check to see if my frames were saved and they were. So even if Terragen crashes, always check to see if your frames were saved.
There is also an option in Preferences (File Saving) to save all renders even if you don't chose "Render All to Disk" or "Render sequence" from the Render Panel.
-Derek
That's a good note. You may want to check your windows logs after this happens and look for what error tgd.exe encountered. I'm going to try and start logging these when I find them.
You can view the logs under Event Viewer, under Windows Logs -> Applications. You can than Ctrl + F and search tgd.exe and try and narrow things down. You'll have access to the type of error, where it occurred, and dumps that can be used to help diagnose issues.
Here's an example log of the most recent terragen crash I had:
Quote from: Application LogsFault bucket 1935929454206262490, type 5
Event Name: FaultTolerantHeap
Response: Not available
Cab Id: 0
Problem signature:
P1: tgd.exe
P2: 4.3.23.0
P3: 5C563A10
P4: ffffbaad
P5:
P6:
P7:
P8:
P9:
P10:
Attached files:
\\?\C:\Windows\SERVIC~3\LOCALS~1\AppData\Local\Temp\FTHA2F3.tmp\fthempty.txt
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERA333.tmp.WERInternalMetadata.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERA538.tmp.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERA545.tmp.csv
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WERA585.tmp.txt
These files may be available here:
Analysis symbol:
Rechecking for solution: 0
Report Id: 2e173674-475e-4460-854b-da1b75244bff
Report Status: 268435456
Hashed bucket: 29ddb3a5f5c64a6e4addcd94e42e1cda
Cab Guid: 0
And a critical error:
Quote from: Application LogsFaulting application name: tgd.exe, version: 4.3.23.0, time stamp: 0x5c563a10
Faulting module name: ntdll.dll, version: 10.0.17134.799, time stamp: 0x7f828745
Exception code: 0xc0000374
Fault offset: 0x00000000000f479b
Faulting process id: 0x2a68
Faulting application start time: 0x01d53dc5fc21846a
Faulting application path: C:\Program Files\Planetside Software\Terragen 4\tgd.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: a7ea3a24-a2bb-4291-81b9-eb554818c2cf
Faulting package full name:
Faulting package-relative application ID:
No dumps are found with these bad errors, and I guess that may be because windows killed the process.