Planetside Software Forums

General => Terragen Discussion => Topic started by: SuddenPlanet on December 20, 2021, 11:58:35 AM

Title: 99% of final pass - Stuck
Post by: SuddenPlanet on December 20, 2021, 11:58:35 AM
Anyone ever do a large render and have it get stuck on the final filtering pass?

My render is a 16,000 x 16,000 perspective render from above (top view but using perspective, not ortho).

I'm using the Mitchell-Netravali filter, and it's so slow (not multi-threaded).

Are any of the filtering methods multi-threaded?

Thanks,

Derek
Title: Re: 99% of final pass - Stuck
Post by: WAS on December 20, 2021, 12:09:33 PM
The final filtering pass are technically layers, overlayed on your final image. That being said, if you're using more than just GISD, I'm pretty sure there are 16,000x16,000 rasterization being generated ontop of the final render tif, and then applied to your image. This is where people that don't have enough RAM often crash when their render is already filling RAM. Additionally, some of the filters like Bloom/Atmo bloom are doing calculations to a degree over a basic image effect like Starburst.
Title: Re: 99% of final pass - Stuck
Post by: SuddenPlanet on December 20, 2021, 12:15:06 PM
OK, I'll just wait and see if it finishes.  I'm only using 75GB of my 128GB RAM, and I've done 16k renders before without issues.

The render itself only too 37 minutes, but this filtering pass has been going 2 hours so far.

I'll try the other filters to see if any are faster.  I assume that Box is the fastest/lowest quality, and they are in order of quality but not sure.

Thanks,

Derek
Title: Re: 99% of final pass - Stuck
Post by: WAS on December 20, 2021, 12:24:08 PM
That does seem a bit extreme, but I'm not sure what the file is like to know. I know sometimes they can hang, like I once did a starburst that was too strong, and it made the render window go unresponsive, and when it finally applied I just had a white blown out image xD
Title: Re: 99% of final pass - Stuck
Post by: Nala1977 on December 20, 2021, 01:15:35 PM
it happens to me a lot of times even on smaller resolutions.
Title: Re: 99% of final pass - Stuck
Post by: WAS on December 20, 2021, 02:08:36 PM
Yeah it can definitely hang even at low resolutions. If there is a lot of shadow I think there is a lot more GISD applied for example.

I'm not really a fan of TG Bloom (though atmo bloom is wonderful), so I often just do bloom in post with levels, desaturation, Gaussian blur and than layer style as screen.
Title: Re: 99% of final pass - Stuck
Post by: SuddenPlanet on December 20, 2021, 02:14:48 PM
Are you talking about Post Process Effects, or Pixel Filtering (Box, Tent, Cubic etc...)?

It's still stuck, so it looks like I'll have to kill it.  I'll let it go one more hour just in case.

Derek
Title: Re: 99% of final pass - Stuck
Post by: WAS on December 20, 2021, 03:22:57 PM
Post Processing. Pixel filtering should be done at render time, not post. You see the result when the bucket is finished.
Title: Re: 99% of final pass - Stuck
Post by: SuddenPlanet on December 20, 2021, 05:36:11 PM
Quote from: WAS on December 20, 2021, 03:22:57 PMPost Processing. Pixel filtering should be done at render time, not post. You see the result when the bucket is finished.
Ah ok then I guess my issue had nothing to do with the pixel filtering.

Not sure why it got stuck, but I had to kill it.  I didn't turn on any post effects.  Anti-aliasing bloom is on by default, so I'll try turning that off.

Thanks,

Derek
Title: Re: 99% of final pass - Stuck
Post by: WAS on December 20, 2021, 07:49:32 PM
Are you using GISD? Or PT?
Title: Re: 99% of final pass - Stuck
Post by: SuddenPlanet on December 20, 2021, 11:27:57 PM
GISD at default settings.
Title: Re: 99% of final pass - Stuck
Post by: WAS on December 20, 2021, 11:55:11 PM
Hmm. But you do mention 75gb of 128. If a GISD pass at same resolution is being applied it could be around the same size in memory, which would be about 150gb so the pass could be fragmented with scratch disk (which could become errored). I am not positive and someone more technical with the code would know better than me.
Title: Re: 99% of final pass - Stuck
Post by: Dune on December 21, 2021, 02:33:42 AM
At least the rendertime itself is extremely short, so you could have another go. Perhaps try two crops...
Title: Re: 99% of final pass - Stuck
Post by: SuddenPlanet on December 21, 2021, 12:40:27 PM
I'm trying some more things.  After doing some more renders it seems like the render itself finishes, but it croaks on the saving out part.
I'm going to try turning off the auto save so it doesn't automatically save an exr of my render to see if that makes a difference.
Title: Re: 99% of final pass - Stuck
Post by: WAS on December 21, 2021, 01:18:51 PM
Maybe try 32bit TIF output instead of EXR? There isn't much benefit to a 32bit EXR as it doesn't contain extra data the format is used for.

Oh, that's right. TG still just does half-float. >.<
Title: Re: 99% of final pass - Stuck
Post by: SuddenPlanet on December 21, 2021, 02:46:47 PM
Quote from: WAS on December 21, 2021, 01:18:51 PMMaybe try 32bit TIF output instead of EXR? There isn't much benefit to a 32bit EXR as it doesn't contain extra data the format is used for.

Oh, that's right. TG still just does half-float. >.<
I'm not sure you are correct on that point about Terragen only supporting Half Float.  Has Matt said that?  Or have you seen that stated anywhere in the Documentation?

I did a ton of HDR Skyboxes for the last game I worked on and was doing a lot of technical image crunching to get the Dynamic Range of Terragen's output down because it was outputting HUGE ranges of brightness in the Sun Light (Sometimes over 100,000).

For example I just did a test render of the default scene, moving the sun down to 15 degrees, and leaving it at the default 5 for brightness.
I then saved the resulting image as 16-bit Tiff (32-bit Tiff not available in Terragen that I can find), 32-bit EXR, and 16-bit EXR (Half Float).

After loading the three images into photoshop and sampling the brightest spot of the sun I got substantially higher readings on the 32-bit EXR.
Here are the Results using the 32-bit Eye Dropper:

EXR 32-Bit: 73777
EXR 16-Bit:  5844
TIFF 16-Bit: 1.0

Does that make sense?

Derek
Title: Re: 99% of final pass - Stuck
Post by: WAS on December 21, 2021, 03:01:39 PM
I mentioned that on line 2, that there is no 32bit float. And I just mistook TG for having such a common format in DCC.

That's where I'm just like "Meh" with some of the logic put into TG. It's backwards. EXR is a waste when it's not even utilizing the standard method of use, incorporating channel information, etc, in one file.

And then we don't have 32bit float TIFF, which would be the proper exporting method here, since we are not utilizing the standard workflow of EXR.

Also stranger yet we have a reader that can read 32bit float TIFF, but not write it.
Title: Re: 99% of final pass - Stuck
Post by: SuddenPlanet on December 21, 2021, 03:28:36 PM
Ah I see.  Well maybe we will get some of those things in Terragen 5. 

On a positive note I did get some of my 16k x 16k renders to finish and save.
One thing I did was turn off GI surface details, so maybe that was the problem.

Thanks for pointing me to potential GISD problems. 

Derek
Title: Re: 99% of final pass - Stuck
Post by: SuddenPlanet on December 21, 2021, 03:50:45 PM
Oh, and I just remembered another reason I don't like Tiff.  Photoshop CS5 complains about not having enough memory to open large Tiff files, even though there is plenty of RAM.
Same file saved as an EXR loads just fine.  Maybe I need to upgrade to Creative Cloud (I'm guessing the latest version might be better at loading Tiff).  Just tired of more and more subscriptions.
Title: Re: 99% of final pass - Stuck
Post by: WAS on December 21, 2021, 04:05:48 PM
Quote from: SuddenPlanet on December 21, 2021, 03:50:45 PMOh, and I just remembered another reason I don't like Tiff.  Photoshop CS5 complains about not having enough memory to open large Tiff files, even though there is plenty of RAM.
Same file saved as an EXR loads just fine.  Maybe I need to upgrade to Creative Cloud (I'm guessing the latest version might be better at loading Tiff).  Just tired of more and more subscriptions.

Photoshop 2021 trial is spoiling me. I don't want to go back... Lol But i can't afford a subscription to all this stuff. No way. I'm not a studio.

I haven't really had much issue except for the typical, files over 4gb won't load, regardless of EXR or not. I think that's a rasterization issue, as Affinity Photo doesn't have an issue.
Title: Re: 99% of final pass - Stuck
Post by: WAS on December 21, 2021, 04:06:53 PM
PS Affinity Photo is a little strange coming from Photoshop, but it's well worth the price for it's 32bit support, and large file support.

I also think it's buy once. I bought it a little over a year ago now, I think, and it still gets updates to new versions from Windows store.
Title: Re: 99% of final pass - Stuck
Post by: WAS on December 21, 2021, 04:11:55 PM
Sorry for another post. I just don't want it to get lost in other off-topic info.

It would be great if TG could export it's GISD pass as a layer, so in situations where GISD generation is too much, you can just do another render for the GISD itself and comp it in post. As far as I'm aware GISD is just doing an AO like effect which could be exported as a layer?
Title: Re: 99% of final pass - Stuck
Post by: SuddenPlanet on December 21, 2021, 06:37:28 PM
Yea, maybe GISD could be added into Render Layers/Elements.

Oh and thanks for reminding me about Affinity Photo.  I bought it awhile back on sale for $25 and was able to convert my 16k tiff to other formats, so that was nice.  Thought I had wasted a render.  :)
Title: Re: 99% of final pass - Stuck
Post by: noahding1 on December 24, 2021, 03:58:03 AM
Increase memory. I have not had any crashes since I increased memory by 64GB
Title: Re: 99% of final pass - Stuck
Post by: SuddenPlanet on December 24, 2021, 11:41:37 AM
Quote from: noahding1 on December 24, 2021, 03:58:03 AMIncrease memory. I have not had any crashes since I increased memory by 64GB

I have 128GB
Title: Re: 99% of final pass - Stuck
Post by: Matt on December 24, 2021, 12:47:32 PM
Quote from: WAS on December 21, 2021, 01:18:51 PMOh, that's right. TG still just does half-float. >.<

Internally Terragen renders colours using 32-bit floats, and when saving EXRs you can choose between 16-bit and 32-bit float.
Title: Re: 99% of final pass - Stuck
Post by: Matt on December 24, 2021, 12:52:32 PM
Quote from: SuddenPlanet on December 21, 2021, 03:28:36 PMOn a positive note I did get some of my 16k x 16k renders to finish and save.
One thing I did was turn off GI surface details, so maybe that was the problem.

GISD can take a while at such high resolutions, but I was still surprised to hear that it was taking hours for you. Was the GISD using default settings?
Title: Re: 99% of final pass - Stuck
Post by: SuddenPlanet on December 24, 2021, 01:10:37 PM
Quote from: Matt on December 24, 2021, 12:52:32 PM
Quote from: SuddenPlanet on December 21, 2021, 03:28:36 PMOn a positive note I did get some of my 16k x 16k renders to finish and save.
One thing I did was turn off GI surface details, so maybe that was the problem.

GISD can take a while at such high resolutions, but I was still surprised to hear that it was taking hours for you. Was the GISD using default settings?

Hey Matt,

Yea, all Default Settings for GISD.  I never did definitively determine if that was the source of the hang at the end of the rendering though.  There were a couple other things I changed settings on like turning off Anti-aliasing bloom, and I might have changed shadow settings on the Sun.  I could more precisely determine the hang source with some more test renders if it is helpful.  

Derek
Title: Re: 99% of final pass - Stuck
Post by: WAS on December 24, 2021, 01:34:21 PM
Next time it happens see if the task manager shows disk usage. If so it may be like I was mentioning and TG is trying to offload its assets to harddisk instead of RAM. I used to use scratch instead of RAM when I only had 4gb of RAM. While it worked mostly for me, it worked in slow motion, and was prone to crashing. But hey with 512gb allocated at the time I was able to do huge scenes that even this PC with 32gb crashes on. Just have no disk space available for scratch.
Title: Re: 99% of final pass - Stuck
Post by: SuddenPlanet on December 24, 2021, 03:26:44 PM
I always watch the Performance Monitor when I am rendering/building in various programs like Terragen, World Machine, etc.

I remember CPU activity was very low, like less than 1%.  No Disk activity, but I did have some Ethernet activity, which I assume was because I was using Remote Desktop to access my PC.

As far as disk swapping, I don't think Terragen or Windows would do that unless RAM was used up, and like I said before I was only at 75GB capacity of my 128GB (Windows 10 + Terragen).

I'll do some more test renders when I get time to see if I can nail this down to a specific setting.

Derek
Title: Re: 99% of final pass - Stuck
Post by: WAS on December 24, 2021, 04:54:32 PM
Yeah but if your project is ballooning that much, and it's making a GISD pass at the same resolution in 32bit or what not, that GISD pass can be around the same size in memory, that's about 150 GB, well over 128GB. It then needs added computation and no doubt memory when it comps the two together.
Title: Re: 99% of final pass - Stuck
Post by: SuddenPlanet on December 27, 2021, 12:19:04 AM
OK well I did a whole day of test renders to figure out the threshold at which renders crap out at the end because of what we thought was a memory issue with GISD.
Windows never shows in the Performance Monitor that memory is being used up, so I am curious why that is.  It feels more like a Terragen render buffer limitation.

I have determined 100% that the issue is related to having GISD turned on, but it's not merely a render resolution issue.
It seems to have more to do with how much highly detailed terrain is in the camera view (I am using 32k heightmaps/32,768 x 32,768 pixels).
I am not doing standard landscape renders pointing at the horizon with a sky and ground.  I am rendering 100% ground from a top down view.

Basically what I did was create a simple scene not much different from the default scene, but I added a Heightfield Generate node to make a terrain instead of using my 32k Heightmap.
I will attach the scene here so you can try it out on a machine with enough CPU & RAM.  I left the Heightfield Generate at 4096 so the scene will load faster, since auto generate is on.
The terrain I generated was 32km with 32768 points, so basically the same as a 1 meter DEM.  You will need to up the heightfield generate to 32768 points to experience the hang at the end of the render.

I then did some 16k renders with GISD off to get a pretty fast 16k render on my 24-core Threadripper.  My test scene renders in about 10 minutes.
I suspected that I could get a 16k render with GISD on to finish if I rendered just a small portion of the 32km terrain at 16k rather than the entire thing, and that hypothesis turned out to be right.

Then I rendered from a top down Perspective Camera with GISD turned on to see at what FOV would cause a render to hang.
Having too wide of a FOV on a highly detailed terrain (32,768 points/pixels) will capture an enormous amount of detail in the render.

I also tried a couple different resolutions in the Heightfield Generate node, and was able to get the full 32km terrain to render with GISD on.
So by reducing detail in the heightmap/heightfield it had a similar effect to reducing the FOV, and the render finished.

Incidentally, when I say the render wound hang at the end I mean it didn't finish.  It didn't actually hang Terragen or cause a crash.
It just got to 99% and stopped, but appeared to still be working on something because the "Stop" button to abort the render was still there.
I was able to close the Render Frame Buffer window and continue working in Terragen.

I guess the bottom line is I don't really need GISD on for what I am doing.  For anything high altitude you wouldn't see the details anyway.  Correct?


Derek
Title: Re: 99% of final pass - Stuck
Post by: WAS on December 31, 2021, 02:01:53 AM
I wonder too if you also played with GISD values. I believe default is 24px? That may be way too big for such tight details. I don't know how it computes but maybe that has something to do with it?

But also, from such far distances, you may not need it. I never notice it actually do anything on orbital shots visibly, though I never do turn it off. But it seems it's not needed. I think this is even seen with distant terrain in haze in normal scenes.
Title: Re: 99% of final pass - Stuck
Post by: masonspappy on January 01, 2022, 04:49:45 AM
I had a problem like this a while back, but don't know if it's applicable here.  My render was getting stuck on clouds (23+ hour render and it still wasn't finished) and I went back and UNticked the "receive shadows from surfaces" box. Rendering took off again, going from mind-numbingly slow to just annoyingly slow.
Title: Re: 99% of final pass - Stuck
Post by: Dune on January 01, 2022, 05:05:41 AM
Yes, I also changed parameters while rendering, and it often works. I pause, then change things that won't be very obvious in the render but take a lot of time. Like reflectivity in certain parts. Uncheck and the render goes on much faster.
Title: Re: 99% of final pass - Stuck
Post by: WAS on January 02, 2022, 03:39:49 AM
Quote from: Dune on January 01, 2022, 05:05:41 AMYes, I also changed parameters while rendering, and it often works. I pause, then change things that won't be very obvious in the render but take a lot of time. Like reflectivity in certain parts. Uncheck and the render goes on much faster.
I always forget the renderer is dynamic like that. Actually could be useful for stuff like what Masonspappy mentioned, get done rendering area you needed receive shadows, and uncheck it where you don't when it gets to it (pausing of course, TG finds it very rude of you to make changes while rendering and may decide to leave you).
Title: Re: 99% of final pass - Stuck
Post by: SuddenPlanet on January 05, 2022, 07:29:11 AM
Quote from: WAS on January 02, 2022, 03:39:49 AM
Quote from: Dune on January 01, 2022, 05:05:41 AMYes, I also changed parameters while rendering, and it often works. I pause, then change things that won't be very obvious in the render but take a lot of time. Like reflectivity in certain parts. Uncheck and the render goes on much faster.
I always forget the renderer is dynamic like that. Actually could be useful for stuff like what Masonspappy mentioned, get done rendering area you needed receive shadows, and uncheck it where you don't when it gets to it (pausing of course, TG finds it very rude of you to make changes while rendering and may decide to leave you).

Wow Dune, I had no idea the Terragen renderer could do that!

I would be interested in knowing how to adjust the GI Surface Detail settings but there isn't much in the Docs.  It just explains the rendering process:
https://planetside.co.uk/wiki/index.php?title=Terragen_4_Global_Illumination#Render_GI_Settings

If there is a thread you remember that explains what the settings do please post.
Title: Re: 99% of final pass - Stuck
Post by: Kevin Kipper on January 05, 2022, 12:51:04 PM
Hi SuddenPlanet,

Be sure to check out Part 23 of the "Terragen for VFX" videos for an overview of the GI Surface Detail settings.  It's about 09:24 into the video.
https://planetside.co.uk/terragen-tutorials-for-vfx-series-5-rendering/terragen-for-vfx-part-23-render-settings-global-illumination/