That's definitely a memory issue. You cannot limit total memory use in an arbitrarily complex scene because there are simply certain requirements for rendering particular scene elements. You can't get around the need to load an object into memory if it needs to be rendered, for example. The best you can do is to reduce the size of the render cache, but this can only save perhaps 200MB on a quad core system (or 300MB on a dual core - there is a minimum of 50MB per core).
64 bit support will be one of the next things we focus on, but we have no release date for it yet. It will be a free update for registered users. In the meantime moving to a 64 bit OS will help you as you can use up to 4GB of RAM, and of course you will be ready for when 64 bit TG2 is available.
- Oshyan