You're definitely running out of RAM. TG2 isn't a 64 bit app, so as Richard said it can only access *up to* 4GB of RAM, even on a 64 bit OS. On a 32 bit OS it's 2GB max, unless you're using the /3gb switch, in which case it's 3.
Generally speaking the default render cache size is fine for most situations. I seldom recommend changing it, and when I do, it's usually to *decrease* it. Significantly increasing it won't usually help render time and is generally unnecessary.
So rendering with 8 cores at that resolution may be a problem. You might try 4, with 400MB cache.
- Oshyan