@mediaaetas: I must say I have read your post multiple times now and I am still not clear what you actually would like to know. It would be good if you could be more precise, and re-phrase your question.
For example, I assume you are using the free version of terragen 2, because you seem to say that you cannot push antialiasing beyond three - at least that's what I hear you say.
AA3 won't give you smooth result, but with the free version you can't go beyond that.
Secondly, how do you define low versus high definition? You probably need to know that again the free version is limited to an image width of 800 pixel.
To render at higher resolution, neither vray or rinos or anyting else is getting you there. You need the Terragen 2 renderer to do the job for you, and you need the "Deep" version of TG2 do render at high definition and quality.
All that being said, I'm just interpreting what I read, not being clear if I am just not understanding you correctly.
Regards,
Frank