Question about setting up a renderfarm

Started by sanaia, June 10, 2019, 05:55:17 AM

Previous topic - Next topic

sanaia

Hello everyone,

at first I'd like to say "Hi" to you all, since this is my first post in this forum. I'm a freelance artist and work for a production company. I'm rather new to terragen (but not that new to CGI). Two weeks ago I started my first terragen project that we need for a musicvideo project. The landscape and animation design so far went well, I managed to get realistic ocean waves and cloud layers too. My first tries w/ the Pixelplow renderfarm were also successfull. So far, so good. But when I compared the farm prices for our upcoming project and made a cost/value estimation based on the TG4 benchmark table and the data I meassured myself, I needed to figure out, that setting up our own small farm (and using pixelplow as backup /accellerator) is way more cost effective.

So at the moment we are thinking about a 5 machine i9-9900k setup, supplementing my desk workstation. We also probably need to use the "classic errosion" which is probably not availably for linux.

So my questions would be:


  • How many licences do we need to order for this setup ?
  • How many render nodes are covered by a single license ?
  • Are native linux binaries available ?
  • Would it be possible to run TG4 under WINE (or "Crossover") in case we need the "Classic erosion" plugin ?

Thanks in advance and
w/ best regards,
Sanaia

cyphyr

Replying mostly because I am in a similar position and am interested in the answer.
Welcome to the forum, glad to see you're progressing fast:)

If you're using Terragen Professional you have 5 render licenses included (these are described as "Additional render licenses with purchase" so I believe you could have a total of six instances of Terragen rendering simultaneously.)
https://planetside.co.uk/terragen-product-comparison/

Without getting into too much personal financial data I'm interested in knowing how you came to your conclusion that it would be cheaper to build your own farm.
i9-9900k are coming in at about £2k each so that's a £10k upfront cost plus extras.

(Actually now that I'm typing it out and rough guestimating it is starting to make more sense) But I'd still like to see your break down of how you decide that it is better to run your own farm that use PixelPlow who are very good value compared to everyone else I have come across.

Also what farm manager were you thinking of using?

Cheers

Richard
www.richardfraservfx.com
https://www.facebook.com/RichardFraserVFX/
/|\

Ryzen 9 5950X OC@4Ghz, 64Gb (TG4 benchmark 4:13)

sanaia

QuoteWithout getting into too much personal financial data I'm interested in knowing how you came to your conclusion that it would be cheaper to build your own farm. i9-9900k are coming in at about £2k each so that's a £10k upfront cost plus extras.

i9-9900k incl. mITX mainboard, ram, case and m.2 ssd is roughly ~1000€. I'm assembling everything myself - so no unneeded bells & whistles. After having tweaked a lot of rendering parameters the cost estimation for the PP-Farm still says ~3k€ for 15min CGI in 1080p - this is just the TG4 stuff. Actually I pushed some test animations through PP. My collegue is working with C4D for the buildings/interriour etc. - this has not even been counted in yet. This is only one project - but we are working on some more. Currently these projects don't have external funding - so we are short on money w/ them.

I intend to start out with 1 or 2 machines and will add nodes as needed - but I need to design it for 5 .. 10. For the moment I need to get TG off my video editing machine, since it is unusable when TG is running.  :-\ My tests and calculations suggest that 3x i9-9900k roughly equal the compute power of PP at powerlevel 0.8. Everything else is simple algebra ;)

Actually, I compared the farm speed against my current workstation (HP-Z230 i7-4770k) and found that PP@0.8 is about 12times faster. Then I run the TG4 benchmark on my machine and looked it up in the benchmark list for the i9-9900k. Everything else is simple algebra ;)

I did the same for TR2990WX, TR1950X and I-97980XE. Based on compute speed and price the I9-9900k showed the best power-to-cost ratio and has the least power consumption (heat production !).

Rendermanager ? I don't know yet; it depends ... All I know so far is that in a optimal case it should transparently integrate my little local farm and PixelPlow and I don't want to pay subscription costs for the rendermanager - since in this case I'd write my own.

w/ best regards,
Sanaia

cyphyr

Quote from: sanaia on June 10, 2019, 10:37:30 AM
...
Everything else is simple algebra ;)
...
Hmm yeess s s ...  ::)

I think I could build an i9-9900K system for about £1200 (a big drop on my earlier £2k estimate but not close to your ~€1k (£890) ), the ram would certainly bump the price up (I'd want at least 64Gb) but I guess for a farm it don't have to be super fast.
This is something I have been wondering about investing in for a while and this is certainly food for thought.

Thanks for the insight
:)

www.richardfraservfx.com
https://www.facebook.com/RichardFraserVFX/
/|\

Ryzen 9 5950X OC@4Ghz, 64Gb (TG4 benchmark 4:13)

Oshyan

As Richard mentioned, each Professional license comes with 5 render nodes (perpetual, not subscription). So if you have a single license already, you should have enough for the mini-farm you're talking about.

A native Linux render node is available, yes, but no GUI.

We do not have a Linux SDK, so erosion is not available on Linux, as I think you know. Running Terragen under Wine is possible, from what we've heard, but it's not something we test for or support. So you'd be on your own there. I've seen a couple discussion threads around the forums here which might help you get it up and running. Not sure how it would affect performance though. You ought to be able to output the Classic Erosion result to a static heightfield after calculation and then just load that at render time to avoid the Linux limitation though...

Let us know if you have any other questions.

- Oshyan

sanaia

I made some progress so far:

I set up a new machine as dispatcher running linux. As rendermanager I'll try DrQueue first, if this fails, I'll try "VVertex Muster9". Since I have to do coding anyway because I want to use PixelPlow for rendering too, I first wanted to check out DrQueue. It's not working out of the box and I had to fix some bugs first to get it to compile and run on Cygwin and Linux. The PixelPlow-Agent I got to work under linux using "CrossOver" (commercial WINE version). Just out of curiosity I tried TG4 too - it's running (but unstable) and only using one CPU so far. Maybe w/ some tweaking multi-cpu usage is possible too, but I don't know yet.

BTW: DrQueue doesn't seem to support TG4 yet but only earlier versions - so I'll have to fix that too. After this I'll try to write a wrapper script that submits jobs to PixelPlow instead of launching a local TG4 process, but the current PixelPlowAgent package is unfortunately lacking a commandline tool  :-\

cyphyr

I would be surprised if PixelPlow (or any commercial render farm for that matter) would allow a command line submission to be scripted.
It would be far too easy for someone to make a mistake and submit a render job that ends up costing a fortune only to be informed of the bill too late and after the fact.
www.richardfraservfx.com
https://www.facebook.com/RichardFraserVFX/
/|\

Ryzen 9 5950X OC@4Ghz, 64Gb (TG4 benchmark 4:13)

sanaia

#7
Actually, on their website they even have a topic in the faq covering this question:

http://www.pixelplow.net/support/command-line-job-submission/

There they clearly state they have one and also explain how to use it. It's just that it is not included in the package - in contrast to what they say on their website. Pixelplow has a budget feature, so it's possible to script it in a way that a budget for a project is set and this then is split among the individual render jobs. If the budget is used up, no further jobs will be submitted to the farm anymore.

cyphyr

Quote from: sanaia on June 20, 2019, 02:45:27 AM
Actually, on their website they even have a topic in the faq covering this question:

http://www.pixelplow.net/support/command-line-job-submission/

There they clearly state they have one and also explain how to use it. It's just that it is not included in the package - in contrast to what they say on their website. Pixelplow has a budget feature, so it's possible to script it in a way that a budget for a project is set and this then is split among the individual render jobs. If the budget is used up, no further jobs will be submitted to the farm anymore.

I stand corrected ... :)
www.richardfraservfx.com
https://www.facebook.com/RichardFraserVFX/
/|\

Ryzen 9 5950X OC@4Ghz, 64Gb (TG4 benchmark 4:13)

sanaia

you were not that wrong - I just got the reply from the PixelPlow support: 

QuoteSorry, but we abandoned the CLI submission tool over a year ago because 1) Nobody used it and 2) We had to make changes to the GUI agent that would permanently break it.  As a result, there is no CLI app, and there won't be one in the future.

Thanks,
Pixel Plow Support

:-\

sanaia

I just contacted GarageFarm about the issue, and they are way more supportive than PP. Depending on the project volume, they offered custom support. Since we also need to render Cinema4D and Blender animations and have some commercial projects in the pipeline, I need a workflow here where I don't have to jump through too many loops. So right now I try to sort things out with GF.

Oshyan

Garagefarm is not licensed for Terragen 4 currently, as far as I can see. Are they suggesting they will render Terragen 4 frames for you?

- Oshyna

sanaia

Indeed, GF provides no Support for current versions of TG, as they communicated in later emails  :-\

Anyway - I dropped the idea of integrating an external renderfarm into my render manager directly. The lack of a commandline submission tool is preventing it.

I'm still fixing DrQueue (open source render manager). It is quite outdated (2015) w/ no support for recent versions of TG and broken Cygwin support. However, now I got so far that I am rendering (right now!) the first animation sequence on a Windows/Cygwin slave. The farm manager is running on a Linux machine and the render queue manager can be started on either system (win/linux)   :)


Oshyan

Sounds good, glad you are working on DrQueue. Will you be contributing those changes back to the public repository?

- Oshyan

sanaia

Yes, I'll do that once it is stable and cleaned up. Currently I' testing TG4 w/ "CrossOver" (WINE). If I can figure out how to set it up propperly, then I could also use the windows version under linux as render client (incl. windows binary plugins). Currently it is working (somehow) but not out-of-the box yet.