Extending Terragen with Windows Virtual Memory

Started by WAS, August 08, 2018, 07:43:20 PM

Previous topic - Next topic

WAS

Since there seemed to be some confusion about Terragen able to use Virtual Memory under Windows 10 memory complex, I did a test comparison. I made a scene with two populations and glassy rocks with mesh displacement, 512 faces. Super dense.

The takes awhile to get going, but once going will fill out all available ram, and than all available virtual memory. Compensating for only having 8GB of RAM. Traditionally, when I had 4GB of RAM, and even on this PC before remembering to enable more space for virtual memory, TG would simply crash or get memory access violations and than crash. I couldn't do large dense populations let alone large objects.

This, as you can imagine, provides a very slow system and you won't want to be doing anything else. But hey, if you're low on memory, and low on cash, it works.

Total amount of Physical RAM: 8GB in two DIMMs

RAM Used by TG (that it was able to allocate) 5.8GB
Virtual Memory TG used: 12.6GB
Total Memory used by the Terragen Process: 18.4gb

I restarted my system before starting the render on the test project, which I think is where that "cached" memory in virtual memory is from. The Windows SuperFetch that studies habits, and gathers startup resources for regularly used programs to start them faster.

How to Manage Virtual Memory (Pagefile) Windows 10
https://www.tomshardware.com/news/how-to-manage-virtual-memory-pagefile-windows-10,36929.html

Oshyan

I don't know who you think is confused about it. What I stated - and what you reiterate here - is that use of Virtual Memory for Terragen should be avoided as much as possible because it *massively* slows down rendering. That's a simple fact. Nobody said you couldn't use virtual memory, or that Terragen would not use any, or anything like that. Just that it's a bad idea unless you absolutely have no other option. Which apparently is the case for you, but in my view the experience of using Terragen 4 in such a memory constrained environment is so sub-bar (and would be for any other major rendering application too), that I would never recommend it.

Your accomplishments within those limitations are all the more impressive, certainly, but you are the exception to the rule IMO. You've already stated elsewhere that long render times don't matter so much to you, which is fine, but they do seem to for the majority of people we hear from.

- Oshyan

WAS

#2
Quote from: Oshyan on August 09, 2018, 09:20:24 PM
I don't know who you think is confused about it. What I stated - and what you reiterate here - is that use of Virtual Memory for Terragen should be avoided as much as possible because it *massively* slows down rendering. That's a simple fact. Nobody said you couldn't use virtual memory, or that Terragen would not use any, or anything like that. Just that it's a bad idea unless you absolutely have no other option. Which apparently is the case for you, but in my view the experience of using Terragen 4 in such a memory constrained environment is so sub-bar (and would be for any other major rendering application too), that I would never recommend it.

Your accomplishments within those limitations are all the more impressive, certainly, but you are the exception to the rule IMO. You've already stated elsewhere that long render times don't matter so much to you, which is fine, but they do seem to for the majority of people we hear from.

- Oshyan

What was said to me is that I don't know how virtual memory works, and Terragen doesn't use it. It's conveyed pretty clearly. Reality is the opposite. Also render times is another falsifiability; they are not that long, nor remotely "massively" slower. I can do a full scene with high GI, max MPD of 0.6 and AA6 with V3 clouds in a couple hours... No different without the modification on a supposed system within specifications of TG4.

Too say it "Massively" slows down render time would mean the minimum specifications are not correct for the application like I have been stating for months, as it struggles on my system in general, even without the modification, even just the GUI is slow in a normal scene that isn't basic without memory near cap. In reality, there's not much a difference in render times. If any, it'd be hiccups, during the actual I/O process of render parts. Assuming, though, Microsoft's memory complex isn't incompetent, the most active bits would be happening in RAM itself while lesser used processes moved to committed virtual RAM, such as when a render is just slowly chugging away on a cell because the rest are near complete.

I'd dvise to do some testing before stating things you haven't experienced. Namely in system requirements from the get go. I mean they're incredibly vague as if not really tested at all when CPU sets should be targeted within the 4-core range. Since speed is also such a seriously detrimental concern as well, what about memory type? It plays a exponential roles in RAM I/O. Someone could be running a slow 4-core with 8GB+ of DDR2-400. Be much slower than what I got going on.

In general, like I am doing, a persons virtual memory may be on a drive, that could be dedicated like mine, with decently fast write speeds. My drive is 1.2-1.6gbp/s and only used for software cache, memory, and temporary downloads as it's not that big. And I'm pretty sure this drive is pretty slow and cheap compared to a standard brand name.

----

Also considering this is clearly aimed at those filling out their RAM, and struggling with crashes, it would probably be advised in there circumstances if they aren't going to, or be able to afford RAM.

As, I mentioned before with my last failed HDD, TG crashes corrupt HDDs, the tiff files become corrupted and the sector eventually fails. Every corrupted file in the file scans were related to TGs temp files. They coudn't even be deleted by chkdsk, let alone manually through Hiren boot tools. This was my HDD used with TG for almost 6 years on a multimedia HP workstation I had dedicated for it that my father got me in 2005. Crashes in general often corrupt open files in a read/write scenario.

Also, again, again, with programs moving large chunks of data to and from RAM, it's advised to have a larger virtual memory space to accomodate overflow. This is advised with movie rendering with large raw frames being processed, in lossless format where 10 minutes of video can be 8+GB, and in stuff like Photoshop. In fact it's exactly why Adobe Developed "Scratch" so people didn't have to edit their system but instead simply enable it on default drive C or pick a drive. I'd say it should be advised in TGs case considering how quickly, and the size of chunks that fill RAM. Scratch even does the same as virtual memory and has for like a decade, and can accommodate for low RAM on mobile devices such as Surface or other art based mobile tablets and laptops.