Hey Dune, The Badger and Matt,
I tried Matt's suggestion to get vector displacement output from Terragen and it works.
First I tried regular scalar displacement, heres the Terragen render:
[attach=1]
and a Maya render of a flat plane displaced with an Exr from Terragen
[attach=2]
and a flat plane displaced with the same Exr in Mudbox:
[attach=3]
And also but not least!
The same terrain with some fake stones rendered from Terragen:
[attach=4]
and the Maya scene rendered with vector displacement on a flat plane:
[attach=5]
and finally a plane in Mudbox, using vector displacement (using Absolute Tangent as the vector displacement type):
[attach=6]
Set up in Terragen as Matt suggested, the trick was to use a Displacement shader to vector piped into a Surface shaders luminosity, but making sure the Displacement shader to vector is taken before the Compute Terrain. Also, making sure the Plane object is sitting at 0 in Y coordinate.
Then used a Render Layer to isolate the plane so the terrain was hidden and did a top down ortho render out to OpenExr.
Works really well!
I used to use the Micro Exporter to get terrain geometry out of Terragen, and I just found a post from Paq about the vertex merging problem, I used a free program called Meshlab to fix those duplicate verts easily and also do a nice decimation, but the meshes from the Mudbox displacement are great, very clean and quads too! (a little on the large side though :-), but you can choose the level of detail you need.
I guess the same process would work with something like Zbrush too, once you've got the vector displacement map.
Here's a link to the Terragen TGD files if you want to have a play:
https://dl.dropboxusercontent.com/u/81475291/terragen_vector_displacement.zipGraham
p.s vector map is rendered at 4k and a little detail is lost, but I like what it does to the fakes stones!