I think that probably there is no more detail in your DEM and that is what makes your mountains look so round (more round than in reality). However, if your DEM is very detailed from the beginning, then that should of course also be reflected in your rendering. I myself have never used the kind of converting (DEM to sgi.file) that you describe so maybe something is happening in the conversion.
Also, looking at your second picture, the detail that is seen actually looks like fractal detail added by Terragen, not actual detail from the DEM.
Regards / Robert