Newbie - help on generating specific worlds from heightfields

[Todd]

Hi All

I'm relatively new to all this (3-4 weeks under my belt) and can generate random worlds and colour shaders populated by objects dispersed across painted shaders quite successfully, however I now want to create a specific custom world and seemed to have entered a twilight zone trying to get it to work...

I want to create a customised part of an earth-sized world which is 2900km x 2900km covering roughly western Europe in tem of Lat/Long.  The heightfield data for the area I want to read from a greyscale TIFF (or BMP or TGA - I have it in Photoshop).  Ultimately I want to use different colour shaders on different parts of the map - eg the snowline at the pole will be much lower than that nearer the equator.  I need a consistent level sea (or at least the appearance of one) covering what would be the atlantic/med sea and various large lakes.

I have Win 7, 4GB of memory (which I migh boost up in the next week or so) and an i7 CPU at 2.8GHz. My PC runs quite reliably....except when I use TG2 (the latest free version - but will upgrade to the bought version to get animation if I can get the darned thing to work!)

The TIFF file to be used for the heightfield is greyscale (white mountaintops, black sea also with rivers mapped out in Photoshop with a 1 pixel wide black brush) . The image measures 10287 x 9360 pixels (24bit depth - which I think is 8 bit x 3 channels?). Its 275MB in size.

The approach (and problems!) :
1) I start with the default project and generate random terrain in Heightfield shader 01.  I have "Flatten surface first" in the Displacement shader unchecked so that the terrain doesn't jut out from the globe (thanks Cyphyr!).  I set it to "Position centre" as this seems to work best when I put the sea on.
2) In 'heightfield generate 01' I have new heightfield checked (not sure if thats important or not - unchecking it seems to make no difference), number of points 1000x1000, size in metres 2900,000 x 2900,000.  Feature scale set to 200m (to keep underlying random hills low so they don't distort MY heights for when I figure out how to add them in later for a bit of variety on the landscape).  All other settings left default. I then "Generate Now", sometimes I get a progress bar when it generates, other times it just seems to adjust the view window without any apparent lag/progress bar in generation- either way I now have a 2900k square on my planet.
3) Now I load in my heightfield.  I select "Add Operator>Heightfield Load" then read from file my TIFF specified above - all settings left at default. 80%-90% of the time TG2 crashes at this point...despite only using about 14% of the CPU (typically only 1 of the 8 CPUs fires up) and only 1.5GB of memory (I have the windows task manager>Performance tab open).  
4) IF I've been lucky enough to load the heightfield it will usually crash when I try and add a lake large enough to cover the whole area.
5) If its managed to get through those two stages, it will usually crash when I try and run a standard erode (default settings) across it, if it doesnt fail then it will crash when I try and alter the erode settings.
6) To try and get around this (I assume the dimensions are just too large for TG2, which is disappointing if its supposed to deal with things on a world scale, but I understand heightfields are trickier/larger than fractals), I have tried various ways of segmenting the TIFF file into 3-5 chunks of varying sizes and then trying to stitch it all together, but no matter what I do, I either get one heightfield load obliterating the previous one (even though its offset dimensionally from the first one so the two should only abut and not overlap, and the guiding cube that shows the extents shows two separate abutting heightfields).  Alternatively when I do do something right and get the two heightfields to show there is an ugly (usually flat or sloped) dip between the two heightfields.  Ive checked at the pixel level in the abutting TIFFs are the same shade of grey at the edges so they should come in at the same height.  Oddly enough I dont get this problem if I load 2 very small test heightfields of only a few shades of grey (say 100pix x 100 pix), BUT even then the two joined heightfields dont have exactly the same length as the two TIFFS if I combine them in Photoshop and load as a single heightfield.  I've spent about 3-4 days trying to look at using blend shaders, and fiddling with blend settings to get the edges to match on larger files but I just can't seem to figure them out.

So - what am I doing wrong, or what is a better way of doing it?  Am I better to generate the heightfield then load the data into a displacement shader?  If so I dont think I can erode it in the same way that I've been doing.

I guess at the end of the day I'd love it if I could :-
(a) load my heightfield in say 4-6  irregularly shaped segments, the final result giving me a neatly stitched together/blended area of 2900sqkm
(b) smooth it so that I get rid of the terracing I get when the TIFF loads the different levels of grey (I'm OK at doing this step),
(c) adjust the height of features in my heightfield (I'm OK at doing this for a single heightfield, and assume I'll have no problem with multipe heightfields)
(d) erode some of the features to give me weathered terrain - I'm OK with this and assume that I can erode only the segments/heightfields that I want to which will be smaller and therefore faster to process/trial and less likely to crash.
(e) apply different colour shaders to the various heightfields (I'm Ok with this)
(f) Displace downwards the river valleys so that they can be below the water table and show up as a lake (there's probably a smarter way of doing this, but I havent got past the terrain bit to be able to experiment with it yet or read up much on the forum...I've tried the river operator and it's just too random - cutting through mountains when there is a lower altitude route.  I know there are a number of other applications I can use, but I'm having enough time battling with TG2 to find the time to learn a another new app!
(g) Populate with trees - I can do this using the free sample trees on this first run.  When (if!!) I've got it working I'll go to the full  TG2+XFrog+Animations option.
(h) Populate with other objects (castles/ruins etc)....should be the same as trees I guess (probably a naive statement!)
(i) Try applying a texture to cliffs etc to get a cool look to it...no idea how to do that yet...way down on the list of priorities.
(j) finally clouds and lighting which I'm pretty comfortable with - but I have no doubt that I'll spend hours/days/weeks reading up on tweaks to get the look "just right"

Apologies for the long post....I'm venting about 2 weeks of frustration here in my cry for help

regards

Todd

[Dune]

That's a long story, and a huge operation. I don't know if this is any help for a start, but this is how I would take a greyscale tiff for eroded terrain. And I wouldn't erode such a huge terrain, but only the closest part, where a detailed render will be done. The rest is distant anyway, and may be 'eroded' with some displaced fractals.

[Todd]

Love your tag-line!

Thanks for taking the time to respond.  I think I can follow what you are doing, however I do have a couple of questions :-
1) Why do you use the image map shader rather than a heightfield load - is there any advantage to doing it that way?
2) The settings on the "heightfield shader 01", are they just the defaults?  I assume its only there because its the started default and the 'container' (for want of a better term) for the heightfield generate...otherwise we could plug the output of the erode into the input of the compute (I'm guessing - I have no idea if that actually works, but conceptually I would have thought it did).

To respond to your other point, I want to be able to zoom into any bit of the 2900sqkm terrain and see what it looks like rather than craft a specific terrain at a given point.  I'm using Terragen to help me write  novel and I thought it would be useful to help me visualise (and therefore describe) the terrain at various points as the characters travel through it - also great for providing illustrations to my prospective readers (If I ever get it finished / published).  Already in designing the world its given me a few plot ideas about where action (eg battles - its a fantasy) could take place and the escape routes the potagonists could take based on the terrain.  This is what makes this project tricky - if I didnt want this level of global detail I'd just start with a random fractal world and scoot over it until I found a bit that looked like what I wanted and tune it from there - believe me, with the faceplants I've had ove the last couple of weeks I would have found life much more easy if I'd opted for that route (and still may do so), but reading snippets from the forumes, what I'm trying to accomplish should be possible - if I develop the skill for it (which is admittedly a big IF 

Cheers

Todd.

[Todd]

To give you an idea of my map and the segmentation, the attachments are a much reduced (jpeg compression low quality, size reduced) image of the map and a colour coded view of the masks I'd apply to create each segment/ecosystem....in theory, each mask would give me a segment that I could load/erode/colour independently.

[Dune]

The tag-line is a some story.... based on DandelO's words regarding a thread some time ago, which has now been erased. More fun for insiders who know what it's about, but fun anyway.

Because the tiff is not a terrain, I don't load as heightfield, but as image map. In the shader I just changed the lineair bit, check it out. I made a view of all relevant tabs, the rest is default. You cannot plug the erode into compute, you should use the 'container' indeed.

I think you're in for a lot (a lot!) of work, but I admire your goal. Having written a novel myself and working on a complete (be it small) city now, I know there's a lot of work involved in both.

I like the map, Europe in very early times...

[Kadri]

... Having written a novel myself ...

I am curious can you please explain a little Ulco ?

[Todd]

Hmm - I set it up as you explained (though using a 10k x 10k TIF), however I seem to just have a flat projection even skimming the surface I cannot see any height items BUTyou can just see the shape of my map on the mini screen.  I've tried fiddling with the amplitude and different plan projections, but with no success .

I have noticed two things (which might be related??) on what you sent :-
1) Your image map shader is called "Image Map Shader 01_1"....dont know how you got that.
2) On Image Map Shader 01_1, you have a setting "Use settings from colour tab...", I dont have that.  Is that something in the paid for version, is there a different shader I should have used that has this option.  As its unchecked anyway, I'm not sure it would make any difference.

The attached is what I have - I cant spot what I've missed on yours apart from the above.

Thanks

Todd

[cyphyr]

Just jumping in ... Why is your image map displacement to -30?, actually why is it sat at all?
Try leaving the image map displacement un-checked (off) and set your displacement in the heightfield shader, the third tab (also un-check "Flatten surface first").

or

Plug the image map shader directly into your Compute terrain node and use the image map displacement as you are (although a value of -30 is a little unusual, normally this would be a positive value, white representing the highest value and black the lowest. This would make the white parts of your image 30m deep in your image.

which leads to mentioning ...

Remember that if the image is a normal 8bit image its height values will be interpreted as meters. A white value will be as many meters positive of negative as you type in on your Displacement "Heightfield multiply" value in your heightfeild node. This is NOT the case with GeoTiff's who's white value is represented internally and can be anything (still multiplied by the same Displacement "Heightfield multiply".)

sorry for butting in ...

Richard

Just re-read your post. Pretty sure the reason Terragen keeps crashing on loading a not particularly large file is that although it is a tiff its not a geotiff. Although it will load a normal tiff these are not what its designed for. GeoTiffs can contain a variety of information including high resolution height data. If you plug your image map shader directly into your Compute terrain you should be fine. Just set your displacement to the height on meters of your highest peak and your imagemap size to the size of the area your mapping in METERS.

PS If the original base file was a GeoTiff it "can" be edited but its not easy ... at all ..

[Oshyan]

Is it safe to assume you are running Win7 x64 and not 32 bit? If it is 32 bit, you could be running into simple memory limitations for a 32 bit app (2GB per app, not the 4GB you might assume from your available memory). Large images like that should, in general, load ok, as long as they're in the right format and there is sufficient system memory.

If you want to go the tiled route the best way would be to get your terrain into GeoTIFF format with some form of at least general "georeferencing" to make sure the tiles align easily. Global Mapper or another GIS app would help a lot with this, though you can do some manual georeferencing in TG as well.

Also consider using a dedicated heightfield modeler/editor like World Machine for your erosion, etc. as not only do they provide more control, but they will be better optimized for handling large heightfields and doing complex operations like erosion. Note however that few applications will really be able to perform operations like erosion on a full heightifield of that size. The memory requirements are large. It is almost certain that, even if not for TG, you will need to upgrade your memory (and hopefully you do indeed have Win7 x64).

- Oshyan

[Matt]

Just jumping in ... Why is your image map displacement to -30?, actually why is it sat at all?
Try leaving the image map displacement un-checked (off) and set your displacement in the heightfield shader, the third tab (also un-check "Flatten surface first").

If you're using an Image Map Shader to generate displacement, then you need to set the displacement parameters. Todd is following Dune's suggestion to generate a terrain from the Image Map Shader, so he would need to use the displacement parameters. However, I'm not sure why he's suggesting that, because images can be loaded directly into the Heightfield Load node.

Matt

[Matt]

If you want to go the tiled route the best way would be to get your terrain into GeoTIFF format with some form of at least general "georeferencing" to make sure the tiles align easily. Global Mapper or another GIS app would help a lot with this, though you can do some manual georeferencing in TG as well.

Not necessary. You can specify your own georeference coordinates for any terrain you load, just by turning OFF "auto georeference from file", turning ON "georeference" and filling in your own corner lat-long values. GeoTIFF (the kind that has an associated .tfw file) is only needed if you want to read the coordinates from the file. Even then, technically you only need the associated .tfw, and the tiff itself doesn't need to be GeoTIFF.

Matt

[Dune]

I cant spot what I've missed on yours apart from the above

Did you hit 'generate'?

However, I'm not sure why he's suggesting that, because images can be loaded directly into the Heightfield Load node.

Todd wants to erode, so I figured this to be the simplest way. I never used the heightfield load, so don't know if you can erode after that.

@Kadri: I started writing a fantasy/SF novel in 1986 (400 pages or so), and kept on changing things until a couple of years ago. I never actually finished it, but know how to. No time, however. A pity in a way, it's a nice story, and I thought of some things in there that were later true problems or solutions in real life (like coma drinking, and growing meat in containers). I got three children's books published however, so I cannot complain.

[stgermain]

I tried to do something similar with a map even bigger (in kilometers, not necessary in pixels) than yours. My original idea was to use erosion to the original terrain (in WorldMachine II demo version, I think), but it turned out that the scale of my project was way too big for that. Later I decided it was enough to only do the erosion to a part of landscape visible in the camera by exporting part of the terrain as a heightmap. I never did though.

Instead of using a heightfield I used my map as a blending shader for several power fractals (single pixel in my map was roughly a kilometer or so.). I used hugin to make a spherical map out of it (even bigger, as it spanned the whole planet surface). By using several maps (one for mountains, one for land height etc.) I got reasonable control over the terrain. Sea was made by a sphere object of the same radius and location as the planet but no displacement. I never had problems with Terragen crashing but it did take a lot of memory once it loaded the maps.

The biggest problem in the end was the spherical projection. My origo was in the north pole, so interesting parts of the map were tilted some 40 degrees or so. I could tilt the camera to compensate, but not he populations (and the heightfield export acted funny also). This finally made me give up. You will have the same problem with populations pointing towards y axis and the local normal to another direction. (In latest version you can use the transform shader to rotate your map though... You could center your map to your camera at the origo.)

I wish you luck with your project! You can work wonders with Terragen if you just have the patience of learning how. Now I just might go and pick up the files of my old project...

[Kadri]

...
@Kadri: I started writing a fantasy/SF novel in 1986 (400 pages or so), and kept on changing things until a couple of years ago. I never actually finished it, but know how to. No time, however. A pity in a way, it's a nice story, and I thought of some things in there that were later true problems or solutions in real life (like coma drinking, and growing meat in containers). I got three children's books published however, so I cannot complain.

Thanks and nice to hear . I hope you will finish your fantasy/SF novel , Ulco 

[Matt]

Todd wants to erode, so I figured this to be the simplest way. I never used the heightfield load, so don't know if you can erode after that.

Heightfield Erode is a heightfield operator which can operate on any other heightfield node (green nodes), whether that's a Heightfield Load, Heightfield Generate or any other heightfield operator.

Matt