Most likely to be memory issues
- Open the task manager, and add virtual memory size to the columns viewed
- Start Terragen with %TERRAGEN_PATH%\tgdcli > tglog.txt
- Open your TGD
- Start your render and watch the memory usage
I experience crashes when the total memory usage gets to around 2.1Gb (exactly where depends on what else is running)
After the crash... open tglog.txt and scroll down and have a look at how may triangles the populator has generated. Given the size of your models, I dare say the number will be pretty big....
There are of course other factors to memory usage, but when you're talking about populations, you need to plan to reserve a fair slab of memory for the objects.
You might also consider using billboard versions of your trees beyond a certain distance from the camera. You can easily combine the two using identical population settings for both and incorporating a distance shader into your population density shader to separate the two. This will help reduce the memory usage.