3DEM is pretty old and no longer updated. I wouldn't be surprised if it just can't handle the job with GeoTIFFs. Loading them natively in TG should work, if they're georeferenced they should align properly. I'm not sure if WorldMachine supports GeoTIFF natively, but like Martin my first suggestion for improving the terrain would be to "swell" then erode, so you might kill to birds with 1 stone of WM can do GeoTIFF. Other than that, other dedicated GIS-type software, both free and commercial, is your best bet, something like Landserf, GRASS GIS (both free), or GlobalMapper (not cheap, but pretty much the best product of its kind for working with real-world data).
- Oshyan