The memory needed to render a 10000x7000 image is probably higher than you think. Terragen renders using image buffers which have 4 bytes per colour channel per pixel. It also renders a 3-channel alpha image in addition to the 3-channel RGB image. Those 6 colour channels require 24 bytes per pixel. For a 10000x7000 pixel image Terragen needs *at least* 1602 Mb (10000 * 7000 * 6 = 1 680 000 000 bytes, which is about 1602 Mb). It would attempt to allocate those in 2 blocks of 801 Mb (801Mb for the RGB image plus 801Mb for the alpha image), and that is quite a large block of memory to ask the OS to allocate in a contiguous chunk. Not only that, other image buffers are needed by Terragen to render individual buckets (tiles), and these use up another hundred Mb or sometimes more. Before any of that can be allocated, the rest of Terragen and your scene file may be using a significant amount of memory already. During rendering, more memory will be needed for other things, most importantly the ray tracer subdiv cache (whose size you can control in the render settings) and cloud acceleration caches.
(EDITED) Because TG2 is currently a 32-bit application, the maximum amount of address space it can access is 4Gb on a 64-bit OS, approximately 3Gb on a 32-bit Mac or 32-bit Windows with the /3GB switch enabled, and only 2Gb on a standard 32-bit Windows setup. Even if you have more than 4Gb of RAM installed on your computer, these limits apply. They are only theoretical maxima, and in practice the OS may not be able to give Terragen all of that memory. (/EDITED)
With all that in mind, 10000 x 7000 is probably very close to the maximum resolution that TG2 is currently capable of on a 32-bit system. On 64-bit systems you can probably go a fair bit larger, but I doubt that you could go much larger than 15000 x 10000. If your scene has imported objects in it, or populations, it will be using more memory before rendering even begins, so the maximum resolution will be lower. Other rendering processes need memory after rendering has started and these may cause rendering errors if there is not enough left after allocating memory for the image.
So the simple answer is that we don't know exactly what the maximum image size is, and it depends on your computer, the scene you're trying to render, and your render settings.
After 2.0 has been released we are going to work on a 64-bit build for TG2, allowing it to go beyond the 4Gb barrier on 64-bit systems. We may also look at improved memory handling for large images, using tiling and other techniques to avoid allocating such large chunks of memory, which will be better for everybody.
Matt
