Trying to learn populations

Started by blattacker, October 05, 2019, 06:36:26 AM

Previous topic - Next topic

blattacker

So, having started to (sort of) get a hang of texturing in Terragen, I'm moving now into populating the world with plants and whatnot. As such, I have a couple questions:

First and foremost, I don't know if this is a problem with hardware or a problem with my process, but the moment I load in a population, the program slows to a crawl. I will admit I am using fairly old hardware, though I do have a decent amount of RAM (AMD FX-6100 processor, 24GB DDR3 RAM, and a GTX 760 GPU, if that matters). If I decrease the spacing to a point where ground cover actually covers the ground, it can take my system up to 5 minutes just to populate. Render times don't seem to be affected much by populations, weirdly, so I'm guessing if it's a hardware issue, my RAM is sufficient, but my processor is not?

Secondly, I saw in a post from, like, 4 years ago that loading multiple objects or variations of objects into a single population was planned to be implemented in the future, is this something that's possible now, or are there better workarounds?

Finally, I usually build my terrains in World Machine, then import the .ter file into Terragen and work from there. As such my terrains are usually 8k squares. I imagine attempting to populate the entire square would be inadvisable (even with a density shader to mask out areas where there shouldn't be foliage). Is there currently any way to, using, for example, a select height mask from World Machine, populate the entire terrain, but only in areas that the camera can see? If not, what would be my best option, in that regard? My first instinct is to try a painted shader covering the entire visible terrain masked out by the select height mask.

In the interests of illustrating the problem, this is the current project I'm working on. There is one population (with three objects crudely exported as one just to test if that works) with...I don't know, several million instances, I think? The population took around 2.5 minutes to process, render times were at 1 minute, 12 seconds. 

Side note: I've noticed when exporting an image from the renderer, the colors are never quite the same when I bring it in to Photoshop to export a png for uploading. Any thoughts/suggestions on that? Specifically, the colors seem a bit faded, and shadows are much sharper. Also, exporting as an exr, I expected to be able to adjust details within shaded areas as one would with a photograph shot in RAW format, but that doesn't seem to be the case. Am I missing a setting, or is this just not possible?

blattacker

Welp, I've just noticed the "Clip to Camera" setting, so there's one question answered.

blattacker

#2
I don't know if this is the ideal way to do it, but I have found that if I change Photoshop's color mode to Lab Color, I tend to get better results when editing.

cyphyr

Hey there.
You seem to be working stuff out just fine. I'm not sure about converting to Lab colour mode. Not something I have ever done and I've not ever seen the need.
Generally I save as 16 bit tif unless I'm going to comp an animation in Nuke in which case I save in 32bit exr.

If you want to get an idea of how your system rates in comparison with other users (and therefore it's ability to handle large populations and scenes) check out the Terragen Benchmark.

If you are finding that the program is slowing down too much you can turn off the display of objects or populations either individually (look in the population Preview mode) or globally via the cube button above the preview panel.
www.richardfraservfx.com
https://www.facebook.com/RichardFraserVFX/
/|\

Ryzen 9 5950X OC@4Ghz, 64Gb (TG4 benchmark 4:13)

Dune

Turning off previews (especially in wire, smooth or textured) for any pop established is wise indeed. Also, place your pop areas in front of the camera, so the outer edge is just where the camera is, and with the same angle as the camera is facing. That way you have the best efficiency in populating. For grasses, only use the front areas, because after a few hundred meters grasses won't be really visible anyway and can be replaced by procedural grass or just shading. You can use a distance shader to gradually decrease numbers towards the end of the pop area, so you won't have a sharp ending line of grasses and a gradual flow into procedural. If you want to populate a square k with grasses separated 20-40cm, you'll have a long wait and maybe a crash.

Oshyan

There are a lot of questions here, I'll do my best to answer them all. I know this might sound like too much, but it's worth considering whether to post separate questions in their own threads. I know you have a lot of questions as you're just getting started though, so just keep it in mind. :D I'll reply in a couple posts here since there are several separate subjects, some of which may get a little lengthy.

In regards to performance, there are a couple things to be aware of here. First you need to be clear about exactly where you're seeing slow down, whether it's in a calculation phase (e.g. "populating"), or if it's in a display/interaction context. They are affected by different things. 

Performance of the default 3D Preview is largely affected by your GPU power, not CPU. Your graphics card is pretty old and won't be able to display a lot of population instances, that's true, however the default object display in the 3D preview is "box" mode which should be able to handle a million instances or so on your card without getting too horribly slow. Believe it or not if you have a decent CPU but your GPU is limited then switching to RTP (Ray Traced Preview) mode can give you faster (and more realistic) previews, especially with large amounts of objects. Just be aware that the *terrain* needs to calculate in the standard preview first (TerraTuts has a good tutorial on the RTP: https://www.youtube.com/watch?v=7oPZt45NmW8)

What I *think* you're actually talking about though in terms of performance is the time to calculate a population, i.e. to assign positions to all of the instances of your objects in a population. That's a process that could take 5 minutes as you say, or even more, depending on how many instances you have and your CPU hardware. And this is a process that is *entirely* dependent on CPU performance. Unfortunately your CPU is also pretty old and comparatively slow, which is probably responsible for the performance issue here.

The amount of RAM you have will limit how *many* objects/instances you can populate, and how big their textures can be, among other things. But it doesn't really affect performance that much, unless you're running into your memory limit and using virtual memory, which will be extremely slow. I don't think that's the case here, so it's probably just that your CPU is not really able to keep up with your ambitions very well. While it was fairly good at the time it was released, it is now 8 years old, which is quite a long time in the CPU world of course. Unfortunately we don't have any FX CPUs in the newer Terragen 4 benchmark, but the older Terragen 3 benchmark does have a few which (including a 6100) might give you a sense of what kind of performance you can expect: https://docs.google.com/spreadsheets/d/1eX9Ltn3_9BjsamA0Pxeflv5AKrjkgViEY8VuetB8e3k/

As you can see the 6100 is pretty far down the list, and again the amount of RAM is not really going to affect performance of that benchmark. Even the fastest FX CPU on the list is only in the middle of the pack, and remember this is an older benchmark that mostly doesn't have the latest hardware (e.g. Ryzen CPUs, etc.). It's not my intention to beat a dead horse here, just want to make clear what you should expect based on the hardware you have.

So obviously an upgrade would be the biggest help, but I know that's usually not a helpful response. :D So with that issue in mind, now how do we address the performance limitations and do the best you can with the hardware you have?

The CPU will affect rendering speed, RTP performance, and any calculations like Populating and Heightfield Generation. So for all of those things you want to try to keep your use confined to only what you need. For rendering that means optimizing render settings as much as possible and avoiding really high resolution renders on your own local machine (you can consider using a render farm for this purpose, e.g. if you want to make a printable image for display). For the RTP be aware that the size of the RTP window affects how demanding it is to calculate (which is not as true for the regular 3D preview). One good trick when using RTP on slower hardware is to use the Crop Region which limits the main RTP calculations to your selected area only, allowing you to focus calculation time where you really need it.

For Populations, it's important to understand that even if a population is masked or restricted e.g. to the camera view, it still calculates based on the whole rectangular area of the population. The mask makes it so that instances don't appear where they are masked-out, but the instance position still needs to be generated and checked against the mask, so it still incurs a calculation time. For that reason the two most important things to control with populations to avoid excess CPU use and calculation time are 1: the dimensions of the population itself, and 2: the density (spacing) of the population. 

Try to confine the dimensions of the populated area to *only* what you need. As Ulco mentioned this can include making a relatively small population for grasses in the foreground, then transitioning to a procedural grass shader in the background. Without doing this you could end up with 10s of millions of grass object instances, which will take a long time to calculate as a population and use a lot of RAM. Even with "clip to camera" you'll still want to avoid a population that for example exists behind the camera. While the clipping will prevent those instances from actually being generated and thus taking up memory, they still need to be tested against the clipping area, as I mentioned above, so you'll find that the generating instances part of populating won't really be faster ("inserting" instances should be though because there are fewer of them when clipped to camera). 

For the same reason you really want to determine exactly what density you need for the effects you want and don't increase beyond that. For close-up grass this may mean a very low spacing, to be sure. But consider that you might be able to use lower density for a further grass population, as one example. 

Another thing to consider is the use of Population Caches. Once you have finalized the density and distribution of a population you can save it to disk as a cache. This will mainly only help to avoid recalculation when you re-load the scene since populations generated within a session are generally not recalculated. But there are still some changes you might make to your scene that do not affect your populations but might erroneously trigger a repopulation upon rendering, and with caching you can also avoid that. Just be sure to recalculate and save a new cache if you make a change to your population or anything that might affect it such as terrain shape, masks, etc.

Note that, as you have found, the number of population instances may take a while to calculate, but once finished it may not actually affect render time that much. It does have an impact, but it's less than you might think.

On to part 2...

- Oshyan

Oshyan

So now let's address the rest of the population questions.

First, multiple objects in a population is a feature we do have planned, but has not been released yet. Expect it in a future update.

For populating your imported World Machine terrains, it's important to know that the terrain resolution doesn't really affect the population calculation time. So populating "the entire 8k terrain" won't really be different from populating a 2k terrain, or a built-in procedural terrain for that matter. Also note that 8k is referring to the resolution, but the *size* of your terrain, i.e. its area or dimensions (how big it is in the world in Terragen), is more likely to affect population time, not in a direct way, but simply because at a given density (object spacing), a larger population area will take longer to calculate. So it's the population spacing and total area that affect the number of instances and thus the calculation time. You could have a 100x100 meter population area with 0.1 meter object spacing and it could take longer to calculate than a 10,000x10,000 meter area with 100 meter spacing. Again it's the total number of *potential* instances that essentially affects the calculation time.

World Machine masks such as erosion outputs can make great masks for populations. You can mask populations by a variety of inputs, anything that outputs color data basically. You just connect your masking data to the Density Shader input. So you can load a World Machine erosion output with an Image Map Shader, then use that as your Density Shader to control where your population instances appear. But remember that this will not actually make the population calculate faster, at least during the Creating Instances phase (which in most cases takes up the majority of calculation time for populations). So don't count on this to improve performance, but it *will* improve realism. :)

You can also mask by Painted Shader, procedural noise (e.g. Power Fractal), etc. Again just plug the shader/node(s) you want to use to mask your population into the Density Shader input. A good one to use is the Density Shader, which can allow you to control a population by Height and Slope, both of which are common effects in the real world.

Side note: you can also use World Machine exports like erosion outputs to mask other things like surface shading. In that case you would use it as a Mask input. Just remember (as with population Density Shader masking), the image will need to be scaled and moved to the correct location to get masking to line up correctly. Images generally don't include scaling information, whereas terrain files (like .ter) often do. So your terrain may show up at the correct size height, but an image loaded into an Image Map Shader probably won't. A good way to quickly fix this is just to check the imported terrain details in the Heightfield Load Statistics area at the bottom of the node, then use the same values for size in the Image Map Shader, and get the Position from the parent Heightfield Shader node.

On to part 3...

- Oshyan

Oshyan

Finally, let's talk about color in your rendered images for a moment.

What you see in the Render View inside Terragen is affected by the controls on the Tonemap tab of the Render node you're using. If you save a TIF or BMP, the effect of these settings will be included in the output image file. So if you're seeing a difference between your render and an image editor for those image formats, that is a bit odd. There may be some kind of color correction being applied by default in your image editor? 

OpenEXR is handled differently and is intended to be a more "neutral" output format. The Tonemap controls do *not* affect OpenEXR output, so it will look quite "flat" in comparison since there is no post contrast or other effects applied. But you're correct that both 16 and 32 bit EXR output should have "raw-like" exposure latitude. However this highly depends on your image editor's support for OpenEXR and 16/32 bit images. Newer versions of Photoshop can handle them decently, but almost any display (monitor) you'll have access to won't really display it right, so it can look pretty bad. Maybe you're already familiar with these issues, but if not it will be surprising and perhaps frustrating. OpenEXRs don't really act like camera raw images. However you can get a reasonable facsimile by opening it in Photoshop then saving as a 16 bit TIFF and bringing that in through "Open As" and selecting the raw formats. Then you can use Adobe Camera Raw on it and get access to all the tools you may be used to.

So if you're seeing lower contrast or other issues in your rendered images *and* you are saving in .tif, for example, then next step would be to share a before and after image here. The rendered image directly as saved from Terragen, e.g. tif, and the output from Photoshop, e.g. png. Then we can get closer to determining if there really is an issue or if it's some color profile conversion going on or something.

- Oshyan

blattacker

Thank you all so much! This all has been super helpful! I'll mostly be replying to Oshyan, as there's a lot there to digest, but here we go!

Working as a system builder myself, I am all too aware that my system is outdated. I'm upgrading (hopefully) within the next couple months, but that's always the case with computer upgrades, isn't it? I'm not sure about what everyone is referring to when they say procedural grass, I'll have to look into that. So far, my methods of making grass have either been populations, or going for the N64 style "This area may be flat, but it's grass colored, so that means it's grass" approach. I did start using population caches, and that's been super helpful! I was worried at first that it may result in a ridiculously large file size, but was pleasantly surprised when it came in at just over 100MB.

I realized while reading your reply that I misspoke. I said 8k squares when I meant to say 8km squares, as that's the default dimensions in World Machine. I usually only output at 4096, since I add some detail to the terrain in Terragen, and I use the shader system, and only very rarely output image maps from World Machine. Though I may start outputting masks more often now that I see how useful they are. I can't believe I missed the density shader when I was looking for something like that. I ended up trying to use a weird surface shader to get altitude and slope controls. I do not recommend that ahaha! I have noticed that image maps brought in from World Machine tend to not load in correctly, though it's easy enough to fix. Assuming the terrain I'm applying it to has it's origin at 0,0,0, I just resize to the terrain size (usually 8km x 8km) and then position from center.

Going to the color, essentially what I'm seeing is the render window in Terragen and a TIF or BMP output of that rendered image don't quite match up. I wouldn't be surprised if Photoshop was automatically converting to Adobe RGB or sRGB or something, though. I have a Creative Cloud subscription, so theoretically I should always be running the latest version. Color accuracy is definitely a problem with my current setup, but I will be buying (budget) color grading monitors as part of the computer update, so maybe that will help. There's only so much calibration you can do on an 8 year old Dell monitor and a cheap AOC monitor that's even older. I'll have to try some of the methods you mentioned though!

Finally, I'd like to apologize for having all those questions in one thread. As you said, it probably would have been better to have them separate, but I didn't want to flood the forums with a bunch of posts in a short time period. Especially when a bunch of the questions I ended up answering myself. I'll leave this reply with the current progress on the project. I ended up just setting the size of the population to the entire 8km terrain, and using some masks from World Machine as well as clipping to camera. The population itself still took around 5 minutes to populate, but 5 minutes to populate and like 3 minutes to render with path tracing (I just wanted to try it out) is a comfortable time frame for me. I went a little crazy on editing in Photoshop so the colors are kind of weird, but the whole thing is still a work in progress.

Oshyan

Looks good to me, aside from the oddness of the cloud (?) in the upper-right. I'm glad you found my responses helpful. And I'm definitely glad you have a computer upgrade in your near future. That will definitely boost your productivity quite a lot!

- Oshyan

blattacker

I do have one more question that I just thought of. It has no bearing on anything I'm currently doing and is more of just a curiosity thing. If I was going to be rendering an animation where I did a fly-through of an area with heavy grass or something, would an ideal method for doing that efficiently be to have the distribution shader be a distance shader with the distance set to a reasonable distance around the render camera, and have repopulate every frame on? Or would it be more efficient to just populate the entire area where the fly-through occurs?

Dune

When you repopulate, you're likely to get more flickering as instances won't be the same in every frame.
Something else; if the further white dots in your latest scene are the same flowers as the ones in front, that is what I mean by populating only front (maybe only 20m is needed), and doing background procedurally (small white dots on greens, with a little displacement).
If you look for procedural grass or fake grass, you'll find some tgc's and advice, but it's merely 1 or 2 stacked fake stones with a small size (0.005) and height of 10 or so, colored or not (you can leave out color if the ground they sit on is colored in greens). Sometimes a default shader with some reflectivity and translucency to color the 'stones' will add more realism to distant grass.
Another method for distant grass would be to use the internal grass, which is fast. For very bumpy terrain, decrease patch size and set rotate to terrain.

blattacker

Oh, that is immediately helpful! I suppose not much can be done for procedural trees, but I'm not doing nearly as many of those, so I can probably just deal with the added processing time for the populations. It'll be helped by the fact that my general foliage populations can be much smaller now! I didn't have much luck finding any clip files in the forums, but after that explanation, it was easy enough to figure out on my own, so thank you!

blattacker

First off, sorry for the double post! After playing around with it, I've noticed that mid-field, the stones don't seem to "pick up" the color shader applied to them. Am I doing something wrong here? I could definitely remedy this by just coloring the underlying terrain, from what I understand, but it's a bit odd to me. I've attached a test render illustrating the problem. Same scene, different angle, and using the built-in grass clumps population. Also, is it natural for the procedural grass to look so flat in the far distance? I haven't messed with the reflectivity or translucency of the default shader applied to them, is that perhaps what's missing?

Dune

If you use a slope restriction on a surface layer (or distribution shader as mask) and the stones are its child, then they'll use the same slope restriction, so the sides won't be colored. You could try using slope restriction with terrain normal as slope key, rather than final normal, the latter of which has a finer/more detailed effect.
And the further away, the finer the 'grass' so you need high MPD (0.6 or 0.8 or so) and a larger render to have some distinction rendered. I would also add some color variation by one or more stacked PF's to those stones (uncheck displacement).