The ultimate Terragen machine(s)

Started by rolland1013, September 20, 2017, 08:38:49 PM

Previous topic - Next topic

rolland1013

So in the past six months Terragen has become an indispensable tool for my company's vfx work.  It started with location plates we were receiving from a new client.  We had instructed them as to what we needed, but what we got back we pretty much unusable.  It was at this point we looked to Terragen.  I took it upon myself to step in and start learning the software, and while I'm in no way an expert yet, I have created several locations/environments/assets that easily pass as the real thing.  At this point I see us looking to Terragen before requesting plates from production.
That being said, it is clear our hardware is in need of an upgrade to better work with Terragen.  Most of our workstations are dual proc Xeon machines with 64GB of memory, Quadro K5000 or 6000 video cards running Windows 10 (our main 3D package is Maya).  About half of our render farm is in worse shape, at the moment I am using the 5 render licenses that came with my seat of the program, but we will be purchasing 60 more render node licenses within the month.  I've done renders on every machine in our render farm and the times are all over the place, the fastest will render a frame in a bit over two hours, while the slowest will render that same frame in twelve.  My question is what is the best hardware set up for a Terragen workstation and render farm box?  Oshyan had previously mentioned the AMD Ryzen Threaddripper and Intel i9 CPUs.  But I'd like to know what would be the ultimate set up for a workstation and render box.  It appears that either solution is a single proc set up, or am I wrong about that?  If possible I'd still want whatever solution to work well with Maya.

Thx,
Niel

digitalguru

There are so many different variables in render farm setups, you have to find the one that best suits your studio. for instance at some places the render farm can share cpus on a single machine with other running jobs, at others they devote the entire machine to a single job ( I guess the former is more about economics of maxing out your farm), obviously with Terragen the more threads you can throw at it, the better, but your render management software might be running another job on the same machine, using say half the threads available and more importantly a significant portion of the memory.

So I think good render farm management software is a must, that can balance out the load on your machines as well as a balance between running Terragen and render jobs from Maya (rendering with Arnold?) and maintaining consistency across your render boxes. Can those slow 12 hour render boxes be put in a pool to only render 2d jobs for instance?


Oshyan

Hi Niel,

First, make sure you have adjusted the graphics card driver settings of any of your machines running Quadros, as we discussed in the support thread previously. If you haven't already done that, then it's not surprising you're seeing highly variable render times. Assuming those settings are correct already and your render times are the best they can be, it would be interesting to know the specific hardware in question. But perhaps not that relevant to your question, if you're looking to upgrade.

When looking at hardware, for Terragen it is generally fairly simple to get the best performance, and then it just comes down to how much money you want to spend for each extra bit of performance. Terragen is currently a CPU-only renderer. The GPU is only used for limited OpenGL display purposes in the 3D preview and node network, basically. So for best performance you just want to get as many CPU resources as possible for your money, and then get as much RAM as you can afford, especially if you're rendering more complex scenes. I would recommend 64GB minimum for render boxes at this point.

With the Zen CPU family from AMD (including Ryzen, Threadripper, and Epyc) now being very similar to Intel's "Core" series (including Xeon's) on a clock-for-clock basis, you can now basically just use clock speed times number of cores to arrive at a roughly equivalent level of performance for a given CPU or system (in the case of multiple CPUs). The one caveat to be aware of is that the more render threads you are using, the more efficiency is lost to "overhead" (inter-thread communications, work distribution, thread scheduling, etc.). The amount of overhead varies for each renderer, and can be difficult to quantify, but hopefully it suffices to say that if you have the choice between more threads at a lower clock speed, or fewer threads at a higher clock speed and they result in the same basic number when multiplied together (e.g. 4 cores at 4Ghz = 16 vs. 8 cores at 2Ghz = 16), then you want to go with fewer threads at a higher clock speed.

So obviously there are multiple factors you have to balance - hardware cost, power usage, heat, space (do you have tons of room to put lots of computers, or do you need them all small and rack mounted, for example), etc. I'll give a couple example scenarios which will hopefully be helpful and get you a good basic idea, and then we can talk more specifics if you need to.

First, I think it's worth talking about new vs. used hardware here. Due to the lack of competition from AMD until recently, Intel was in a period of pretty slow improvement for a number of years. Performance would improve maybe 5-10% annually with each new CPU release. Since a lot of companies replace hardware on a time basis for reliability reasons, rather than strictly on a performance basis, it meant that there was a lot of older used hardware that was still very high performance, perhaps 80-90% of today's CPUs but at 50% or less of the cost. You can still find a lot of used dual Xeon machines on eBay and other sites that are a really good deal for hardware that's 3 or fewer years old and still performs really well. To outfit a render farm on the cheap it can be a great way to do it, especially if you buy a really reliable workstation model/brand and/or you have in-house hardware expertise and can e.g. swap out an SSD or RAM should the need arise.

Anyway, if you want to buy new, here are the options as I see them:

Starting at the best price/performance ratio, which assumes space is not an issue (because we're talking about single CPU machines). I'd recommend a Threadripper in that case. Ryzen 7 1800x is 8 cores at 3.6Ghz, which is 28.8 "Ghz equivalent" (this is the term I use for these simplistic calculations, and keep in mind they are not precise measures of expected performance, just ballparks). The 1800x costs about $500 right now. The top of the line Threadripper 1950X is 16 cores at 3.4Ghz, a slight drop in per-core clock speed, but twice as many cores, 54.4 Ghz equivalent, so obviously a lot more powerful than the 1800x. It's about $1000, so twice the price of the Ryzen CPU for not quite twice the Ghz equivalent, however when you factor in the cost of the rest of the components (making total system cost less than 2x) and inherent limits of space, it's still a better deal to get 1.9x the performance for less than 2x the price.

So a single Threadripper 1950x machine with an SSD and basic graphics card (because TG doesn't use it for rendering) would be a great render box, and will be hard to beat for price/performance ratio. In comparison, the Intel i9-7920X is $1200, and 12 cores at 2.9Ghz = 34.8, much worse than Threadripper (not to mention the motherboards are more expensive, I believe). The 7900X is a slightly better deal at around $1000, 10 cores at 3.3Ghz = 33. So slightly slower, but cheaper and remember we talked about efficiency, so 10 threads at a higher clock is probably going to be better, especially given it's only 1.8 "equivalent Ghz" slower (and $200 cheaper). But both are a good deal slower than Threadripper, so no contest really.

The upcoming Core i9-7980XE - launching this month -  is going to be 18 cores at 2.6Ghz = 46.8, but it's $2000 (msrp). In leaked benchmarks it appears to be outperforming Threadripper on Cinebench, but Terragen is a bit less efficient with high thread counts than C4D, so you're still probably better off with Threadripper, even putting aside the doubled cost of the 7980. That being said, one complexity with the i9 CPUs and - to a lesser extend Ryzen and Threadripper - is that they can overclock themselves, sometimes on all cores at once, depending on thermal conditions. So if you cool your CPU and overall system really well, these Ghz values might be well exceeded, which could account for some of the i9's performance lead over Threadripper (aside from architectural differences, of course). The 7980XE can theoretically do 3.4Ghz on all cores in the right conditions, which brings it up to 18x3.4 = 61.2 equivalent Ghz. Still if you are rendering for many hours on each system you can't necessarily count on that overclock being maintained, so I always feel it's best to consider base clocks, where Threadripper wins handily.

Now if you need more performance with the same space footprint (i.e. a single computer case), you'd go dual CPU. This is going to get you more performance still, but at a notable cost increase. It's not the most performance or the most money you can spend (that honor goes to quad CPU or octa CPU machines), but due to threading efficiency it's probably the most you'd want to spend on each machine, unless space is at a real premium.

Unfortunately for AMD, their Epyc line is pretty low clocked, with lots of threads, so not the best for Terragen performance. A single socket Epyc 7551 is 32 cores at 2.0Ghz, or 64Ghz equivalent, but since you're using twice the threads of Threadripper, actual Terragen rendering performance might not be much greater. And that's also a $2100 CPU. If you step up to dual CPU Epyc, for the same performance level in the 7501, it's $3400 *per CPU*. The Epyc line tops out at 32 cores at 2.2Ghz = 70.4 per CPU in the 7601, but that's even worse at $4200/CPU. Still, Intel's Xeon line doesn't do any better at full retail price. If you buy used, as I mentioned above, you can get some slightly older but still very high performance systems for a bargain, which would easily compete with Epyc for price/performance. But if you're buying new, I'd suggest either Threadripper (a real bargain for the performance), or a dual Epyc 7501, which is going to give you 64 cores total, at 2.0Ghz = 128Ghz equivalent, which is crazy. Honestly with that number of threads (including hyperthreading it's 128 on a dual CPU setup like that!), you'd probably want to run 2 parallel instances of Terragen for best performance.

With both Epyc and Xeon you can go up to quad CPU, but then you're looking at $10-20k or more per machine, and it just doesn't seem worth it. :D

So bottom line: Threadripper is the sweet spot, best price/performance for new hardware. Get a dual Epyc if you need more and are buying new, but you'll save a lot of money buying a used dual Xeon in that case. For dual CPU I'd definitely recommend the used Xeon route. I have an HP Z820 workstation with dual 2.9Ghz 8 core CPUs in it, plus 128GB of RAM and dual SSDs, and it works great. It's in the top 10 of benchmark results for the old Terragen 3 benchmark and it was about $1800 on eBay. It even included about 6 months of remaining warranty support, with on-site tech visits.

Let me know if you have any questions.

- Oshyan

bobbystahr

Though not what I'm trying to do machine wise, that was most informative for a future build Oshyan, thanks for taking the time...great support.
something borrowed,
something Blue.
Ring out the Old.
Bring in the New
Bobby Stahr, Paracosmologist

rolland1013

Hey Oshyan,

Apologies for my delayed response, and thanks for your detailed answer.  I'll be going over it with my systems guys.  They're also getting me the specs for our render farm, I'll post it when I get it. 

Digitalguru, the render farm management software we use is Deadline v.9

Niel

silentgod

Possible info' on next EPYC processors.

https://twitter.com/CPCHardware/status/925348852147908608

Ne le répétez à personne, mais les CPU EPYC de 2e gen d'AMD auront 64 cores, 256 Mo (!) de L3, 8x DDR4-3200 et 128 lignes de PCIe 4.

( Second gen' AMD EPYC cpu will have (upto) 64 cores, 256mb of L3, 8*DDR4-3200 & 128PCIe-4 lines.)

;D

Oshyan

64 cores at 1Ghz each? :D Really though I don't want more cores as much as I want *faster* cores. 32 cores at 3Ghz would be great.

- Oshyan

bobbystahr

something borrowed,
something Blue.
Ring out the Old.
Bring in the New
Bobby Stahr, Paracosmologist

rolland1013

Since I first posted this question almost a year ago I'd like to ask if there are any new recommendations for workstation hardware?  We're looking to buy a couple new computers in the next couple of months, and I'd like to spec them out for optimum TG performance.

Thanks,
Niel

WAS

Quote from: rolland1013 on August 15, 2018, 03:07:42 PM
Since I first posted this question almost a year ago I'd like to ask if there are any new recommendations for workstation hardware?  We're looking to buy a couple new computers in the next couple of months, and I'd like to spec them out for optimum TG performance.

Thanks,
Niel

Really don't think there is enough people with the software with dedicated built machines to really test hardware as it comes. I think you can take from earlier statements with newer hardware. For example, more cores/threads vs lower clock speed doesn't necessarily mean you'll be having a "blast" of a time, and you might go with something with less cores/threads and high clock speed. RAM, I mean, is sorta just something you want to be on top of. Fast high quality RAM is only going to benefit scenarios.

bobbystahr

Quote from: rolland1013 on August 15, 2018, 03:07:42 PM
Since I first posted this question almost a year ago I'd like to ask if there are any new recommendations for workstation hardware?  We're looking to buy a couple new computers in the next couple of months, and I'd like to spec them out for optimum TG performance.

Thanks,
Niel

Have you looked at the AMD RYZEN family...that's where I'm heading I think. Would love their ThreadRipper but a bit beyond my pension heh heh heh
something borrowed,
something Blue.
Ring out the Old.
Bring in the New
Bobby Stahr, Paracosmologist

WAS

Quote from: bobbystahr on August 15, 2018, 08:25:11 PM
Quote from: rolland1013 on August 15, 2018, 03:07:42 PM
Since I first posted this question almost a year ago I'd like to ask if there are any new recommendations for workstation hardware?  We're looking to buy a couple new computers in the next couple of months, and I'd like to spec them out for optimum TG performance.

Thanks,
Niel

Have you looked at the AMD RYZEN family...that's where I'm heading I think. Would love their ThreadRipper but a bit beyond my pension heh heh heh

Yeah it seems that threadripper fairs nicely with Terragen, to say the least. At least judging by what little feedback we've seen. Can't wait for terragen to have it's own benchmark statistics either in TXT or watermark form to go along with renders. Some sort of "Advanced" checkbox to hit or something.

bobbystahr

#12
Quote from: WASasquatch on August 15, 2018, 11:41:58 PM
Quote from: bobbystahr on August 15, 2018, 08:25:11 PM
Quote from: rolland1013 on August 15, 2018, 03:07:42 PM
Since I first posted this question almost a year ago I'd like to ask if there are any new recommendations for workstation hardware?  We're looking to buy a couple new computers in the next couple of months, and I'd like to spec them out for optimum TG performance.

Thanks,
Niel

Have you looked at the AMD RYZEN family...that's where I'm heading I think. Would love their ThreadRipper but a bit beyond my pension heh heh heh

Yeah it seems that threadripper fairs nicely with Terragen, to say the least. At least judging by what little feedback we've seen. Can't wait for terragen to have it's own benchmark statistics either in TXT or watermark form to go along with renders. Some sort of "Advanced" checkbox to hit or something.


https://www.tomshardware.com/reviews/amd-ryzen-threadripper-2-vs-intel-skylake-x,5727.html?utm_source=th-newsletter&utm_medium=email&utm_campaign=20180815-th

I'm considering this:
AMD TR 1920X 12 / 24
3.5 / 4.2 speed 64 L3 CACHE
64 (4 to PCH)
Quad DDR4-2667
180W $399 $33 price per core
something borrowed,
something Blue.
Ring out the Old.
Bring in the New
Bobby Stahr, Paracosmologist

archonforest

Holly Cow! The 32 core thread ripper is a f... beast. 
Dell T5500 with Dual Hexa Xeon CPU 3Ghz, 32Gb ram, GTX 1080
Amiga 1200 8Mb ram, 8Gb ssd

digitalguru

#14
Hey bobbystahr,

If you're going with a Threadripper, make sure you get some decent cooling, those CPUs get hot

For reference, I got a Corsair H80I for mine (all-in-one water cooler with a single exhaust fan) and I don't think it's quite enough.

Also, the Threadrippers are larger CPUs so make sure any cooling solution covers the die size adequately.

QuoteHolly Cow! The 32 core thread ripper is a f... beast. 

I reckon it could probably do the Terragen benchmark in under a minute - and probably calculate next months weather forecast at the same time :-)