I don't believe file format is the limiting factor with importing terrains quality. I think it has more to the data resolution, and how many meters it covers.
Seeing as all a heightmap really is, is a bunch of values between white and black, black being low, white being high values, and the pixel represents one datapoint of height. Example: a 100x100 pixel hight map wont hold much quality/datapoints, especially when stretched across something like 10000x10000m in terragen. However, 1000x1000 will look good, higher the resolution, better it looks.