The terrain looks crappy because of the ray detail multiplier setting.
Go inside your rendernode and open the "render subdiv" node. Inside you'll see the ray detail multiplier set to 0.25.
This means that for ray traced elements like water transparency and terrain the detail setting will be multiplied by 0.25.
That's the reason the terrain looks worse than expected and is also the reason why, by default, underwater features look triangulated.
Increase it to 0.5 or 0.7 and see if it is more to your liking. You can set it to 1 max, of course, but be aware rendertimes increase.
edit: I'm wondering if this multiplier also affects RTA?
For RTA it's difficult to determine the settings and it requires good understanding of the pixel sampler.
It's good to know how many samples you require for a 'smooth' result without RTA.
If it's a huge amount (hundreds of samples) then you likely need to make good use of the adaptive AA pixel sampler and relatively higher AA.
Be aware that in case of RTE it also affects your terrain.
So like Oshyan said, if you feel exploring raytracing the atmosphere then go for RTA instead of RTE.