Errors with large TERs and TIFs

[bigben]

I've normally avoided TERs larger than 4097x4097, and TIFS larger than 6000x6000 pixels. I found that using larger files, while not necessarily reaching RAM limitations still caused an increase in unhandled exception errors. 

While preparing my latest terrain set I decided on testing the current version with an 8194x8194 terrain instead of tiling 4 x 4097 terrains.  RAM usage shouldn't be an issue as I've had no problems with my Grand Canyon terrain set which consists of 8 x 4097 terrains.

Loading the terrain for the first time is OK, but then trying to remove the path from the filename (or re-selecting the file) causes an unhandled exception error.  I also set up my masked water surface, leaving the connection of the TER to the duplicate heightfield shader until last.  TG crashes as soon as the last connection is made (load terrain to vertical adjust). 

I also have a greyscale 8194x8194 image map in this file although I've been able to make numerous connection changes from this without any problems to date, so this seems to have improved with the new version.  For now I think I'll try tiling 4 terrains and see how much difference it makes.

[Oshyan]

I presume these are problems you haven't seen and can't duplicate with smaller files? Are you certain there's nothing else unusual about the 8194 terrain? I wouldn't think a larger size should have that affect, although certainly it is bound to break down at some point (memory issues being the most likely culprit at that point).

- Oshyan

[bigben]

I presume these are problems you haven't seen and can't duplicate with smaller files? Are you certain there's nothing else unusual about the 8194 terrain? I wouldn't think a larger size should have that affect, although certainly it is bound to break down at some point (memory issues being the most likely culprit at that point).

- Oshyan

Making the same node network with smaller terrains doesn't have any problems. The total memory usage isn't overly high (around 1gb for the attached file), and as far as cramming data into TG2 I've worked with 9 x 4097 terrains and 2 x 4097 image maps (greyscale).  To repeat the test I took out the 8194 image and replaced it with a version cropped to 4097x4097.  There is also a 4097 RGB image map (blue marble texture)  in this project with a total memory usage of just over 1Gb.

The attached screengrab shows a workaround for creating the lake surface using the original terrain with a small negative displacement ("Displaced rough terrain" group). This is a workaround because it uses a terrain with fractal noise added.

[attachthumb=#1]

It will crash when making connection 1 regardless of whether connection 2 has been made or not.  It's too fast to see any change in memory usage in task manager.  I wouldn't really have expected 8194 terrains to be too much of a problem as they're still "only" 131Mb.

Just tried something else that may help isolated the problem. Instead of making connection 1, I connected the load heightfield node directly to the heightfield input of the duplicate heightfield shader (3) and it didn't crash.  I then replaced the adjust vertical node by connecting the heightfield shader to the displacement shader (4)... same effective result (displaced terrain without fractal detail) and no crashes.  Perhaps it's an issue with heightfield operators on large terrains?

[attachthumb=#3]

Made a new TGD and just added the terrain, duplicate heightfield and adjust vertical shaders. All connected OK but total memory usage was about 400Mb more than the TGD with image maps and lake model above. That increase on its own wouldn't have been enough to cause the crash but it does look like it may be close to the culprit.  Reloading both versions verified that memory usage mentioned is consistent.

I have an adjusted model that works now, so I'm happy. If there's anything else you want me to test just drop me a line.

[bigben]

Just applied the revised water model to my Grand Canyon test (http://forums.planetside.co.uk/index.php?topic=4746.0) and memory usage (RAM+VM in both posts) dropped from 950Mb to 780Mb at the 40% mark of the preview render. This is a fairly sizeable drop given that it contains just a single 4097 terrain (32Mb).  I can squeeze in some more plants now   

[Oshyan]

It makes a lot more sense that it might have to do specifically with heightfield operators rather than a simple load and render of larger terrains. The operators have to perform algorithms on the entire heightfield and a very large one could easily overrun the algorithms I imagine. Something to look into. Thanks for the report.

- Oshyan