Rendering is a process that involves many systems, many of which need to use RAM with varying amount depending on many factors. It won't be possible to quantify all of them here. But I'll start with some of the most important ones I can remember off the top of my head.
All of the following will use RAM, so these requirements add together. The list is not exhaustive by any means, so actual RAM use will be even higher.
1) Final image, including render elements.
RAM is proportional to the image size (width times height). It's also proportional to the number of render elements (approximately). Most render elements use 12 bytes per pixel. So a 1000x1000 image will use 12 Mb per render element. RGB and Alpha elements are always rendered. So a 1000x1000 image will always use at least 24 Mb. A 5000x3000 image will use at least 360 Mb. A 5000x3000 image with 10 render elements (including RGB and Alpha) will use about 1.8 Gb. Some combination of render elements that don't need 12 bytes per pixel may be packed together so the total may be slightly lower.
Final image RAM use is *not* affected by anti-aliasing (AA) settings.
2) Buckets while render is in-progress
RAM use for subpixels in a bucket is proportional to the number of pixels in the bucket, plus some necessary padding/overlap pixels. That is then multiplied by the square of the Anti-aliasing number. It is also multiplied by the number of render elements. Additional buffers are used for Depth of Field, GISD, anti-aliasing bloom, and a depth buffer for occlusion calculations. Each of those additional buffers is proportional to the number of pixels in the bucket plus padding pixels, and proportional to the square of the AA number. GISD buffers may re-use render elements already in memory or may require additional buffers. So it's hard to define exactly how much they will use. There will be other requirements I've forgotten to mention.
With threaded rendering there are usually more than one bucket rendering simultaneously. So you can expect RAM use to increase with increasing number of threads.
As long as "allow auto reduction" is enabled in the Bucket Controls (and it is by default) then the number of subpixels in each bucket should cap out at 1 million subpixels (not including padding/overlap between buckets), so even with very large images or very high anti-aliasing the RAM use per bucket should not climb excessively. However, very high AA values will still use more RAM than low AA because of the padding/overlap area of each bucket, whose number of subpixels is proportional to the square of the AA number and also varies with other factors such as anti-aliasing bloom, DoF method and pixel filter.
RAM used by buckets during rendering is freed when the buckets finish.
3) Anti-aliasing bloom, if enabled, requires additional buffers that are proportional to the size of the image (in addition to anti-aliasing bloom buffers per bucket). These buffers are freed up when the render finishes.
4) GISD, if enabled, requires additional buffers that are proportional to the size of the image. Which buffers are allocated depends on what render elements are already available (because they were enabled in the GUI), so GISD may or may not increase RAM usage. Any allocated buffers will be freed up when the render finishes.
5) 2d motion blur, if enabled, requires additional buffers to be allocated temporarily when the render has almost finished, to perform a post process. Additional motion vectors need to be generated during the render, if the user hasn't already enabled motion vectors as a render element, but they can be freed when the render finishes if the user didn't enable the motion vectors element.
6) Subdiv cache. The RAM used for the subdiv cache should never exceed the "size of subdiv cache" parameter in the render settings. Depending on number of threads, scene complexity, lighting complexity, image size, render detail and other settings in the "subdiv cache settings", the actual RAM use of the subdiv cache may reach the prescribed limit or it may stay below. If you enable "preallocate subdiv cache" then it will always use the prescribed amount of RAM. This RAM is freed when the render finishes.
Please note that we don't recommend changing the "size of subdiv cache" parameter without seeking advice from us first, if you have a particularly challenging scene. In general you should stick with the default which is set when you start a new project (assuming you use our factory default project). On the 64-bit version it varies with the number of cores detected on your system (we also recommend that you use the 64-bit version). With 8 detected cores (e.g. a Core i7 processor) the default is 1200 Mb. With 16 detected cores the default is 1600 Mb.
7) Many other rendering systems. There are other ray-tracing data structures that use amounts of RAM that depend on the complexity of the models in the scene, caches of various sorts, and many, many smaller components that all use RAM that may depends on various parameters. There are probably some big RAM hogs that I've forgotten to mention.
8 ) The scene in memory. Objects, populations, shaders (especially ones that load images), heightfields and heightfield operators, etc. all use RAM before rendering even starts. However, populations that are already populated and objects/images/heightfields that are already loaded into Terragen should not use a lot more memory when rendering starts - the renderer just uses the data already loaded.
9) The 3D Preview has some memory requirements of its own, before rendering even starts. Closing all 3D Previews before you start a render can often free up a good chunk of RAM. This can vary with scene complexity and shading modes, and to some extent the number of cores detected in the system.
10) Other components of the system and general application overhead.
I haven't covered everything, and I may have forgotten some important factors. Generally, there isn't really a simple formula for estimating the RAM use of a render, not even roughly. However, a big factor is image size (width times height) multiplied by the number of render elements. Perhaps equally important are the complexity of your models and the number of instances in your populations. But each of these factors are, for the most part, independent uses of memory, not compounding factors. Anti-aliasing shouldn't usually be such a great burden on RAM, but there may cases where it does. Of course AA does greatly affect render times in many cases.
Matt