Taking it beyond the limit. (populations)

Started by cyphyr, March 11, 2010, 08:45:24 AM

Previous topic - Next topic

cyphyr

I am, once again, at a Terragen limit, or so it would seem.
My current project, a tribute to Avatar's Pandora planet, is running into memory problems.
In order to get populations to work on the near virtical cliff surfaces I'm having to use them at high densities, I'm up to 31 populations (shareing object markers) and I must be about two thirds of the way to completion. Maybe only another 15 or so populations to go :)
So I know I've strayed into the badlands of extreme terragen but I cant see another way to get the densities I need. Each Floating Mountain has to have its own set of Populations (7 of them). Each tower (well the ones close to the camera) also has its own high density populations (3 each in this case as the towers also share wide scale populations.)
When rendering I'm getting unfilled bucket errors, raytrace errors and forced appilcation closure.

So two questions :)
1 Any sugestions about population optimisation?
2 Hows the 64bit version comming on and once its finnished is this the sort of scenario it would excell in?

Cheers

Richard
www.richardfraservfx.com
https://www.facebook.com/RichardFraserVFX/
/|\

Ryzen 9 5950X OC@4Ghz, 64Gb (TG4 benchmark 4:13)

John-117

#1
Hello, nifty scene i did not know you could do the floating mountain thing in Terragen.  As far as 64 bit Terragen I don't see anything on their home page about that and even if they did make one (which I would hope they do) you will still need a 64 bit OS to run it.  As I'm new to Terragen I cannot really help with the populations however, (and you may already know this) if you are running a 32 bit operating system you can only access a maximum of 4 GB of ram (which is very easy to run low on) and frequently your graphics card "shares" some the 4 GB you so might have even less usable ram.  With a 64 bit system you could at the very lest double your ram to 8GB (depending on what OS edition you get you can go way higher then that to) and fix your "running into memory problems" as most 32 bit programs will running on a 64 bit OS fine (there are a few exceptions of course; anti-virus, 32 bit drivers and so on).  Best of luck on your project.

Zairyn Arsyn

they are making a 64bit version - its in the works, and any news/information on the 64bit version has been shared here, on this forum
Quote from: John-117 on March 11, 2010, 10:04:51 AM
Hello, nifty scene i did not know you could do the floating mountain thing in Terragen.  As far as 64 bit Terragen I don't see anything on their home page about that and even if they did make one (which I would hope they do) you will still need a 64 bit OS to run it.  As I'm new to Terragen I cannot really help with the populations however, (and you may already know this) if you are running a 32 bit operating system you can only access a maximum of 4 GB of ram (which is very easy to run low on) and frequently your graphics card "shares" some the 4 GB you so might have even less usable ram.  With a 64 bit system you could at the very lest double your ram to 8GB (depending on what OS edition you get you can go way higher then that to) and fix your "running into memory problems" as most 32 bit programs will running on a 64 bit OS fine (there are a few exceptions of course; anti-virus, 32 bit drivers and so on).  Best of luck on your project.
WARNING! WIZARDS! DO NOT PREDICT THE BEHAVIOR OF OTTERS UNLESS YOU OBEY BIG HAPPY TOES.

i7 2600k 3.4GHZ|G.skill 16GB 1600MHZ|Asus P8P67 EVO|Evga 770GTX 4GB|SB X-FI|Antec 750W
http://zlain81.deviantart.com/

Hetzen

Hi Richard, can't wait to have a look at what your scene looks like rendered. As for populations on steep surfaces, have you thought about using displacements looking like vegitation rather than actual plant objects. If you mix the two, I reakon you'll make something quite convincing. I used distorted stone populations in my NWDA entry last year to mimmick a hill side forest as well as trees/bushes along the field edges, which in your case could free up some memory, but I'm sure you could also use Perlin Billows noise too. The moss on the stones was Perlin Billows, with a bit of tweaking, you could enlarge the techneque with more displacment and a colour adjust shader in there to make seperate clumps.

http://forums.planetside.co.uk/index.php?action=dlattach;topic=6672.0;attach=18015;image

Henry Blewer

I would see if I could use a modeler program to combine the separate objects into groups. It won't reduce the polygon counts, but it may help reduce the population counts.

Honestly, I would have run out of memory some time ago with this population setup you have. I am a t a loss for any good advise.
http://flickr.com/photos/njeneb/
Forget Tuesday; It's just Monday spelled with a T

Tangled-Universe

Can't you render it in seperate passes? It seems some mountains could be rendered separately.
Also, you could try rendering it in a lot of crops, which is crappy because of GI etc. etc., but it is a "solution".

Kadri

#6
Richard , you want to make this maybe in one pass...
But in the film they used separated passes too like Tangled-Universe said .
You can use LOD techniques for the objects but for the population count i think you can use modified objects .
Low quality objects that you made  from the high ones and it would be good  by bringing 2-3-5 or how much you can of them together
and use this unified ones the farer you get on the image. Or billboard  like ones if you want of course.

Kadri.

cyphyr

Thanks for the feedback guys :)
Doing a multi pass render would be ideal, if I actually knew how. The issue with an MP render is separating out the different layers. The floaters will have vines connecting them to the ground and will also have clouds shrouding them in some areas. The clouds will only partially obscure the floaters and the populations resting on them. Everything overlaps foreground to background and side to side so I cant figure out how to break the image up properly. Rendering in crops is of course an option but as has been said many times it is fraught with problems, GI especially.
I hadn't thought of making new objects groups, that might work well for distant large scale areas and indeed billboards is another god choice, possibly combining the two may help.
I'm rendering again with slightly modified settings, basically I've knocked an octave off every Power fractal, hopefully to ever so slightly simplify the displacements.
I'm also now only using the default number of threads (min=0, max=16). Previously I had set minimum cores to 8 to force the i7 to use all its cores but this seems to be the cause of the "raytrace errors".
Cheers :)
Richard
www.richardfraservfx.com
https://www.facebook.com/RichardFraserVFX/
/|\

Ryzen 9 5950X OC@4Ghz, 64Gb (TG4 benchmark 4:13)

Tangled-Universe

Yes, forcing minumum of 8 threads to let the hyperthreading kick also requires the minumum of 50MB/thread of memory. "Ideally" you would use 100MB/thread, so that would already chew up almost 1 GB of the 3 available with 32-bit TG2.

I'll try to think over this more, because I really want to see this finished :)

Martin

Kadri

#9
Richard , did you or anyone tried  to use a flat polygon (or a plane like surface) for crop like rendering ?
Say you put a flat polygon ( surface ) near the render camera and obscure the right side of the view .
Do TG2 count the objects and such behind this surface (as ram usage )? Is there a way for this ?.
Maybe if this is possible there wouldn't GI issues too as you would simply put the surface on the other side for the other render part
and everything on the render camera would be the same. In theory !

Of course there would be some render overheat because of the used surface area . But nothing is perfect...ehmm  :)

Just talking loud. But i think someone had used this if it was possible a long time ago !

Kadri.

cyphyr

Ok this bit has always confused me  ::)

I have 6GB of ram at my disposal and I'm running win7 64bit but of course Terragen is still 32bit so it can only see 3GB of my ram.
I think I get that bit lol. When I force the number of cores up to 8 I get a little pop-up recommending I up the Sub Divide Cache to 800MB, (which gives me 100mb per thread, yes?), I do so and for the most part everything renders ok. Except, (have I figured this out right?) if I'm also loading a vast number of models and populations they load into an adjacent part of memory quickly filling up my remaining 2.2GB of ram. And that's when the trouble starts.
Or, ... .. .
Are the models loaded into the Sub Divide Cache and therefore I could make it much bigger, say 2400MB, and use up to 800MB per thread (for 4 threads or 400MB for 8 threads)?
:)
Scratchin my head
Richard
www.richardfraservfx.com
https://www.facebook.com/RichardFraserVFX/
/|\

Ryzen 9 5950X OC@4Ghz, 64Gb (TG4 benchmark 4:13)

Tangled-Universe

For the sake of easy-ness let's say TG2 can use 3GB of memory (I've used up to 3.3GB in rare cases)

8 threads x 100MB/thread = 800MB
3GB - 800 = 2.2GB left for rendering the GI pre-pass and rendering, including objects etc. etc.

To split the workload I'd load the tgd into two instances. Give those a max threads of 4 each and then render different crops with a bit of overlap.
Yes, there will be GI seams, but they aren't as awful as some time ago, especially with good overlap.
Anyhow, this way you already save 400MB / instance, leaving ~2.6GB for the rest. Effectively 2x2.6 since you're using two instances.

Martin

Dune

Hey Richard, are all those pops in the same area? Otherwise only load the pops in the 'to be rendered part', render it, load other parts' pops and render that part. Then stitch it. The billboards or 'procedural trees' are indeed an option, and another option is very low poly trees. I am currently making some, and will make them available (for free) later today or tomorrow. Just very basic things of 180 kB per object that look ok in the distance. Some testing to do before I upload.
Interesting to see your final product!

---Dune

Oshyan

This is indeed a challenging scene and sounds very interesting and cool. There are a lot of questions here, I'll try to answer as best I can.

TG2 as a 32 bit app can use up to a theoretical maximum of 4G of memory. In practice it never quite gets there, I've seen it get up around 3.7GB though. 64 bit will definitely help with a scene like you're tackling here.

Yes, the render caches are included in this number, so the larger your cache, the less memory you have for everything else. Thus, though TG2 itself will warn against it, *reducing* the cache size, even down to 50MB/thread, will be advantageous for rendering large scenes. It will render slower, but this is a sacrifice you're going to have to accept for now, given the limitations of TG2. In fact that ties in to another possible strategy...

Which is to just render using 1 core. Yes, much slower! But you avoid a lot of memory use. To do this, you would set min and max threads to 1. The idea of rendering 2 crops with e.g. 2 threads each may be the best balance of render time and resource use.

You should also make sure you're using "Crop to Camera" options in populations, unless you're going to be animating (god forbid, hehe).

Combining objects probably won't help that much, it will limit your distribution options and may not have a big impact on memory use, depending.

Using lower geometry and, especially, lower resolution texture images for objects can definitely help. If none of these plants are in the foreground you could probably even get away with *no* textures (except alpha of course, and even that could be lower resolution), and just use colors with power fractals. You'd still want to use the UV Maps to separate trunk from leaves though.

I think that will be good for a start. Let us know if any of that helps!

- Oshyan

Kadri

#14
Quote from: Oshyan on March 15, 2010, 02:33:34 AM
...
Using lower geometry and, especially, lower resolution texture images for objects can definitely help. If none of these plants are in the foreground you could probably even get away with *no* textures (except alpha of course, and even that could be lower resolution), and just use colors with power fractals. You'd still want to use the UV Maps to separate trunk from leaves though...
- Oshyan

Welcome back Oshyan  :)

In my topic about the problem with 256 color images i was aiming for this .
There was more of the very big image usage in my mind . Maybe not so much , but it could help here too in theory .
http://forums.planetside.co.uk/index.php?topic=8750.0
I don't know if this has any practical use in TG2 because it doesn't work for now !?

Cheers.

Kadri.