Natural Watersheds for Rivers

[ejgodwin]

Here's my challenge, and I hope I can explain it clearly. I have a map of an invented world for a novel I'm planning to publish, and I'd like to create scenes from the story using Terragen in hopes of generating interest on web sites, etc. I've converted a hand-drawn map to a heightfield using L3DT, and after some painstaking effort managed to get a pretty accurate representation on which to import a ter file into TG3 and go from there.

My problem is the rivers. I've used displacement via an image shader, which works to some extent, but the underlying slope from the source of the river to its mouth into a lake or ocean isn't consistent. I've tried exporting into a bitmap to adjust in Photoshop, but the height range is so big that the grayscale variations at that level are far too subtle. (We're talking fluctuations less than 10 meters in some places.) I could use the heightfield generate to port it back into L3DT for additional editing, but L3DT's tools are limited, and it would take a very long time. Adding to this time suck is that the heightfield is very large, 50MB at 730x550 km, and the 3D editor in L3DT just drags at any detail level high enough to achieve what I'm after. If I "go with the flow", in other words use erosion techniques and create rivers that way, I introduce geographical differences that have a huge impact on the plot structure.

Two ideas I have in my head, and I'm not sure if either is possible.

1) I can split the rivers into separate image masks for displacements. But I still have the same problem: how to adjust it so it's a smooth slope from the source of the river to its mouth. Another factor is that some of the rivers wind back and forth, even doubling back in some places where the slope is extremely small, like the Mississippi river does. The latter I can tackle by using a combination of techniques, I think, splitting the river shader into segments, and those winding rivers don't really play a part in the story. How to get a smooth slope from start to finish is what's baffling me at the moment.

2) I'm wondering if there's some way I can take a portion of the underlying terrain, using the wider watershed for each river as a basis and introduce a slight slope from one end to the other. This would seem to be the more natural approach. I would still have to deal with a few variations in the terrain, and it might introduce unnatural transitions at the edge of the watershed areas, but those are challenges I could handle in L3DT with far less effort than what I'm facing now.

Hope I've explained this sufficiently. I'm open to other possibilities, of course. And I hope to get to the point where I can share what I've learned and not just beg for help all the time!

[Dune]

Interesting challenge. What I sometimes do is (like you mentioned) use the basic not too rough terrain as input for the water shader line. By masking out the water area, you can raise the actual terrain by subtle displacements, thus keeping the wet areas smooth and slightly lower.  For distance, even just a reflective shader would do, masked to cover the river areas.

[ejgodwin]

Hmm, a reflective shader ... I'll have to look into that. If it works, maybe there's some way I can combine that with a distance shader to introduce a gradually increasing negative Y displacement. I'll play around with that and see where it goes.

[bigben]

From real world data I usually produce 2 images as references for building masks.
One is a slope map, which is useful for masking lakes (exported from GlobalMapper but you may be able to do the same thing in World Machine)
The other is a flow map (exported from World Machine), loading the terrain into WM and applying a small degree of erosion. For a single terrain file I'll export the eroded terrain, but for tiled terrains i just export the mask.

With real world data there are normally small glitches that have to be fixed by hand so I build a mask specifically for rivers and lakes in Photoshop.  Lakes can be isolated by using the magic wand selection with a very small tolerance from the slope map.
Rivers can be a little trickier, but increasing the black level on the flow map is usually a good start. Once you've narrowed down the flow map to what looks good for your rivers, bring the white level down close to the black level to produce a more defined mask.

In TG, I run each of these masks through colour adjust shaders to produce masks of slightly different sizes. Even if you have a bitonal mask image, the antialiasing in TG produces a nice gradient around the edge of your mask. Increase the black level to reduce the size of the mask, or decrease the white level to increase it.  The smaller mask is used for displacement, and the larger mask is used to restrict the distribution of the water, ensuring that it returns below the terrain but stops shortly after, minimising the amount of water to be rendered.
Not sure if this was my latest... http://www.planetside.co.uk/forums/index.php/topic,1164.msg11687.html#msg11687

[ejgodwin]

Thanks, bigben. I actually downloaded your tgd before, but I need to take another look at it. There's actually two learning curves in TG: the interface, and getting my head around all these complex concepts. I've got the first one down pretty well, still working on the second! 

[Oshyan]

What you're trying to is fundamentally a very challenging thing, no matter what application you're using. It reminds me a lot of what the Middle Earth DEM Project (ME-DEM) has been working on literally for years: http://www.me-dem.org/ or http://www.me-dem.me.uk/
They actually worked with a coder who created an entirely application (partly) to generate realistic rivers, watersheds, etc. and then reflect that data in realistic changes to a heightfield. It is challenging work, to be sure. IF you don't want to do it all by hand. If you do, you just need a good terrain editor and a lot of time.

Perhaps the ME-DEM folks, if still active, can provide some input...

- Oshyan

[ejgodwin]

Ah, my greatest fear realized. But that's a good thing, as I don't want to spend countless hours on something that's not going to pay off. I'll just concentrate on rendering scenes without rivers for now and keep an occasional eye out for a solution. Thanks for the feedback.

Bigben: Just so you know, I tried the river test tgd method, and though I got marginally better results, the problem still exists. It was a learning experience to some level, though, so thanks for your help.

[ejgodwin]

Found a possible solution to this problem, but I've hit a snag, and maybe one of you node gurus can help.

Here's my strategy. It occurred to me that a large part of my problem is trying to solve this "river smooth slope" challenge all in one project. (This could be due to my ignorance, but even so it helps keep things clearer in my head.) So I created a seperate project for just one river, and successfully created a flat heightfield with a negative displacement for the riverbed. I also limited the coordinates so it barely encompasses the river, not only to minimize the impact on the overall heightfield, but also in hopes of adding some sort of slope alteration in the future (NE to SW). Right now I'm leaving the overall heightfield of the river tgd project flat and the river displacement exaggerated just to see if this will work.

River Project: A simple vector displacement shader, using an image shader as the vector function. 3D preview looks like this:
[attach=1]
I do a heightfield generator, making sure the coordinates match, and save as a ter file.

Destination project:
I add the generated heightfield to the terrain node flow, using the full "world" heightfield as an input. This is what I get:
[attach=2]

So I have a possibility here, as the geographical placement works, but I'm stumped as to why the river displacement doesn't show up in the destination project. Something tells me it has to do with the height range. If I can get it to work, sloping the river heightfield from source to mouth and adding fractal detail to the surrounding terrain (inverse masking it by the same image shader), I have a real possibility to make this work. Once I have all the rivers added, I can add a final heightfield generator to create a ter file, pretty it up a little more in L3DT, and get what I need. From there I add lake displacements, which are far less of a challenge as there's no slope involved.

River tgd attached. I couldn't attach the image shader because it's a tif file, and the destination tgd requires resources that are way too big.

Any thoughts?
[attachurl=3]

[ejgodwin]

Tried the following: changed the heightfield generator in the river project for a larger height range (0-10000), tried a heightfield merge in the target project, it always ends up flat like in the screen cap above. I've watched tutorial videos about the heightfield merge function, etc, but I admit I'm still stumped on exactly how that works.

I'm stubborn about these things sometimes (I'm a programmer), but I just can't get rid of the notion that I'm only missing one piece of the puzzle. Whether anyone has suggestion or not, thanks for the help.

[Dune]

If you call a tiff (or whatever) in an image map shader, there's no need to use the vector displacement shader. You can use the internal displacement tab and just hook the image map shader before the compute terrain. Or, with dispalcement in the image map shader implemented, add it as a child to a surface shader (all before compute terrain), or output it (without implementing the displacement in the image map shader, if you need that for some reason) to the function input of a displacement shader, or to the displacement input of a surface shader, or to the masking input of a surface shader and use the displacement offset. All sorts of possibilities. Adding it as a child to a surface shader gives the additional plus of extra masking possibilities, and using the offset and displacement tabs of that surface shader separately, e.g. if you want only part of the whole image map shader displaced differently (maybe a waterfall, or extra gradient, or warp, etc).

[ejgodwin]

Thanks, I'll give that a go.

[ejgodwin]

Success at last!

First, I used a simple shape shader to mask the overall heightfield from the area I wanted to create a flat plain, then added a heightfield load from the river image to the same coordinates. Tried all sorts of settings in the number of points, etc, still didn't work. Then I tried resizing the image to the same number of pixels to match the size in meters.

Viola! Turns out that the problem was the image itself. The original was way too small (524x258). (Don't ask me why it worked in the separate river project.) I could have used a smaller size than that, I imagine, but at least it gives me the greatest detail capabilities for editing in PS.

Now to slope it, add some fractal detail, and of course reduce the exaggerated displacement.

[attach=1]

[bigben]

For smoothing the displacement you need some gradation in your mask image.  This can be done a number of ways.
For painting rivers from satellite imagery I found a brush width of 2-3 pixels to be the minimum needed. With such a narrow "river" the antialiasing provides a fair amount of smoothing. Using a larger brush size requires decreasing the hardness of the brush, or alternatively blurring the image. It's easier to have a mask image that is too smooth, as it's pretty easy to tweak with a colour adjust shader in TG.

Here's a sample image and TGD using different colour adjust shaders to make different masks for surface shaders and displacements.   Left side of the mask image is just drawn with a brush in Photoshop. Right half is painted and blurred.  It is possible to adjust the image to the point where the left half becomes a series of small circles, which s why I blur the image.

[attachimg=1]
[attachmini=2]
[attachmini=3]

[ejgodwin]

Thanks, bigben. I took a look at your tgd, and achieved the same results by blurring the river mask:
[attach=1]

I'm ok with the results so far, as I plan to eventually bring this into L3DT and use the fractal tool to tweak a few areas. as well as blend the edges of the shaders. What's slowing me up right now is how to export all this into a standalone heightfield (ter) file. I have to do this, as the heightfield is huge (730x550km), and with all these shaders it's already slowing the 3D preview down, let alone any renders. I have lots more rivers to process, so I thought I'd just export a ter file and start another project, using the same techniques for each river until I was done.

I know how to do the heightfield generate, using the last  displacement shader as the shader input, but I ran into a problem with the "Number of Points" setting. The river in the sample image is only 35 meters wide, and when I used a setting of 5000x3400 (numbers aren't exact, but they were proportional to the heightfield), the resulting output didn't have enough resolution to preserve the river displacement. When I bumped it way up to insane numbers like 70,000 (yes, I was willing to wait), TG3 crashed right away.

I saw another thread about using two cameras and a displacement distance shader to export to a raw image file, so I was wondering if that would be more successful in preserving the detail I've achieved.

[Oshyan]

The Heightfield Generate approach ought to be the most effective and accurate. 70,000 pixels to a side is just enormous though, a multi-GB terrain file (9GB I think?). It's probably a memory issue, which won't be solved by using a Distance Shader, or really through any other export method. I would try exporting a smaller area around a river section, increasing the resolution with each test until you get the definition you want, then extrapolate the needed resolution for the total area. I suspect this will be a challenge though, for *any* application to deal with, either export or import. What had to be done at the Middle Earth DEM project was use lots of tiles...

- Oshyan