Render Farm service and cloud computing for terragen ?

[Oshyan]

It sounds very interesting so far. I especially like the idea of being able to prioritize users in the queue so you could use it as your own render manager as well as for other people's stuff and basically just leave it up all the time.

One issue that arises, both out of the idea of prioritizing your own work and also just generally, is what happens with longer renders. Will there be limits on maximum single render time? Or maybe each user can set their own limits? Let's say someone starts up an animation that takes 1 hour for each frame and they're half way through a frame (30 minutes in) when they decide they want to render something of their own. Does the existing render and 30 minutes of render time just get killed and lost, or is it paused, or do you have to wait for it to finish? 1 hour isn't even necessarily that long for some renders, and without limits I could see people just cranking the render settings and submitting jobs that will take hours or days. So I guess my point is to be wary of (and build in limits/defenses for) abuse of the system, both intentional and accidental.

- Oshyan

[neon22]

Yes its an interesting problem. I hope it will become self limiting and its also one reason why I have tiles.
I intend the render client user to be able to say when the rendering process can run. I typically use my CPUs and only want to allocate them to the community in (say) the evening when I go to sleep (hopefully). Also if the batch portion does its job I will have many choices to make informed decisions about param settings available to me when I start the next days work.

- So you set the start time and indicate the likely end time (you generally know when you leave the computer but not accurately when you will return)

- You manually choose when to stop the renderer. So if you arrive and a frame is partway through rendering - its up to you to determine to throw the render or not. The more accurate you are with your timing indication, the less you will waste your own CPU resources. I would suggest that if you are being paid for a frame and it was 3/4 through and you were early to your system - then maybe you would let it run to completion. YMMV...

- When asking for a frame the system indicates how long until the expected end of service is. If the time is "soon" then the server will only send you short frames (statistically derived on average). This is to prevent sending an hour long frame down 10 minutes before you think you will return and stop the renders. Of course - if there aren't any short frames in the queue, then you get a long frame and we all gamble...
In this way we try to minimise the wasted time.

As a job submitter - If you submit a job and you get very few frames back you have a couple of choices. You obviously want more frames. So you can choose to tile the image, or you can try to reduce the rendertime.
Tiling will get more optimal use of the community resources. (Your own machine will reassemble the tiles). The issue of GI artifacts between tiles has attempted to be addressed but I am unsure as to its final state and the applicability of timing and optimal tile size... Maybe we will do trials and find out...

Ultimately its frustrating if you never get any frames back because they take 6 hours each. So modify your behaviour so you get frames completed.

[Oshyan]

I think you have some good ideas for addressing the issues I mentioned without putting in hard limits or being too intrusive. It's great to see you've got all that in mind already. There's a lot of potential for a system like this. Thanks for spearheading the development!

- Oshyan

[cyphyr]

All sounding really great, just one issue that springs to mind. How would you handle imported assets, both from a licensing and purely practical way. Many of my scenes are not purely procedural and rely on displacement maps, these "can" although not by any means always, be up to a gig in size, more usually down in the 200k region. How would you propose sending these across a network and secondly when imported assets are used such as X-Frog models how will you get about the licensing issue since the animation will be rendered on machines that do and do not have a license?
Cheers
Richard

[Tangled-Universe]

Good questions Richard, I second these.

Another possible "problem" is the renders itself. Rendering animations is fine, but I couldn't stand it at all if I'd notice that the project uses stupid rendersettings which makes things utterly slow. In my eyes it would become "useless" if I simply know it could be much faster.

Good documentation is critical in regard to this and maybe the current rendersettings sticky thread is a bit outdated with the new raytracing options for atmosphere for instance. Also, somehow, people don't seem to find their way to it anymore. Which is strange, as it's clearly a sticky thread and should ring a bell to everyone visiting the forums for the first couple of times.

I'll think about a (re)new(ed) rendersettings guide with more practical examples.

[neon22]

... one issue that springs to mind. How would you handle imported assets, both from a licensing and purely practical way. ... displacement maps, ... up to a gig in size, more usually down in the 200k region. How would you propose sending these across a network and secondly when imported assets are used such as X-Frog models how will you get about the licensing issue since the animation will be rendered on machines that do and do not have a license?
Cheers, Richard

Size - the program parses the tgd, captures all referenced files, zips them into a dedicated directory along with a modified TGD pointing to relative paths for the assets. This gets ftp'd up to the server and redistributed to render clients. Its the smallest mechanism I can think of.

The licensing issue is a great question. Others address this problem by adding a license to the submitted file.
E.g. copyright or one of the CreativeCommons licenses. Basically by submitting it you are saying it is (say) copyright and not to be copied. All users who render have to agree to abide by the copyright rules and not use the assets. E.g. a choice about not participating in copyright scenes, or agreeing to abide by the copyright.

In practise any IP protection is lost except by "agreement". Anyone could look inside your files and see how you did things, or get hold of assets you may heve licensed yourself. In practise sending the same information to a commercial renderfarm exposes the user in the same way but you have a more legally sturdy framework to apply legal redress.

I suspect that this mechanism may not be strong enough for some people and they will choose NOT to use the community renderfarm...

... Another possible "problem" is the renders itself. Rendering animations is fine, but I couldn't stand it at all if I'd notice that the project uses stupid rendersettings which makes things utterly slow. In my eyes it would become "useless" if I simply know it could be much faster.

Yes this could be annoying to you. However if you did look inside and see this - you (or someone else) might send them a note - and overall everone would improve.
But many non-"animation" and non-"big image" renders will flow past unseen - if it's all working smoothly.
Hopefully the end result will reward positive effort and encourage people to get "better" at these aspects....

[digitalis99]

Neon, did you ever get anywhere with this?  I'm toying with the idea of starting a renderfarm service for TG2, since I already have a healthy chunk of gear.

[neon22]

I kept on it for a while but it lapsed when I had some paid work.
Must admit to letting it go for too long. I clearly still need it and I'd like to get back and finish it.
Perhaps I'll have some time over the Holidays...

Thanks for asking.

[NRJ]

Hey! Any news on this? I'd be happy to donate some cycles and help with the renderfarm if possible. I'm currently taking part to the renderfarm.fi but I don't use blender so I'd much rather support tg2 renderfarm!

-NRJ

[digitalis99]

We are starting a render farm service in direct competition to the others that are out there now.  When fully operational, we'll be better than any other farm out there in every aspect.  We're trying to go beta by June, with public release hopefully in July.

[NRJ]

Sounds great. Do keep us updated!

[digitalis99]

You can bet I will.