TG2 Render farms

Started by jwdenzel, December 21, 2006, 04:55:16 PM

Previous topic - Next topic

bigben

#30
I tried a quick job on one computer. It looks very promising...

The job automatically started but it only rendered a default scene (but the correct frame numbers). This may be due to calling tgdcli via the real path rather than using the environment variable. (this happened to me prior to setting the environment variable too) Given the impact setting the environment variable had on TG0.9 I'm guessing that some of the programming of the CLI requires the variable to locate program files.

Opened a command window and manually typed a command to render frame 100 of the same project using network paths and it's running at the moment. I'll let it finish to see if it correctly saves the file (I had to specify that in the command as the local path in the TGD didn't exist)...  This worked

Tried a job using a mapped drive, same result (correct frame, default tgd)

Definitely suggest fixing TG command with environment variable for now rather than user input. You might also want to prompt for an output folder/image prefix to add to the command line to override the internally saved output path.

The command I used that worked on the remote machine (for frame 100) was:

%terragen_path%\tgdcli.exe -p \\host\RenderFarm\Project\project.tgd -r -f 100 -o \\host\RenderFarm\Renders\frame0100.bmp

I'd already considered path issues with the TGD for rendering on other computers and I placed all of my files into a single directory.  I'm sure some scripting guru could whip up a script to prepare a TGD for output by parsing all of the filenames and making a copy of all of the files in a single directory, and stripping the path names out of the TGD.

... oh... and TG remained hidden but then that is the default as soon as you specify -r

Dark Fire

Quote from: bigben on March 21, 2007, 09:56:11 PM
I'd already considered path issues with the TGD for rendering on other computers and I placed all of my files into a single directory.  I'm sure some scripting guru could whip up a script to prepare a TGD for output by parsing all of the filenames and making a copy of all of the files in a single directory, and stripping the path names out of the TGD.
I considered adding the ability to move all the files needed into a single folder to software I wrote for TG 0.9, but I eventually decided to drop the idea. T2TP is forcing me to consider that option again. Stripping out the file names shouldn't be too hard, but it would take quite a while to write because I would have to be absolutely sure that no other data is stripped out.

Quote from: bigben on March 21, 2007, 09:56:11 PM
The job automatically started but it only rendered a default scene (but the correct frame numbers). This may be due to calling tgdcli via the real path rather than using the environment variable. (this happened to me prior to setting the environment variable too) Given the impact setting the environment variable had on TG0.9 I'm guessing that some of the programming of the CLI requires the variable to locate program files.

Opened a command window and manually typed a command to render frame 100 of the same project using network paths and it's running at the moment. I'll let it finish to see if it correctly saves the file (I had to specify that in the command as the local path in the TGD didn't exist)...  This worked

Tried a job using a mapped drive, same result (correct frame, default tgd)

Definitely suggest fixing TG command with environment variable for now rather than user input. You might also want to prompt for an output folder/image prefix to add to the command line to override the internally saved output path.

The command I used that worked on the remote machine (for frame 100) was:

%terragen_path%\tgdcli.exe -p \\host\RenderFarm\Project\project.tgd -r -f 100 -o \\host\RenderFarm\Renders\frame0100.bmp
I'll start working on fixing that as soon as I can. The annoying thing is that, as far as I can remember, being able to set environment variables makes running Terragen with anything other than a normal priority more complicated.

The network path experiment could be made more interesting - I found with TG 0.9 that if the network connection was lost at any point during the render, Terragen would save the output file when it had finished rendering, but would not close and continue to use the CPU as much as it could.

Thanks for showing me exactly what works - when I release the next beta I will be a lot more confident that Terragen will be launched correctly than I would have been if you hadn't supplied me with that information.

Quote from: bigben on March 21, 2007, 09:56:11 PM
... oh... and TG remained hidden but then that is the default as soon as you specify -r
That's good. At least something is working.

Conclusion

I am going to put your entire post in my notes on the next beta, since it contains so much useful information that I simply couldn't obtain with the free version. I am not yet sure when I am going to release the next beta - it will certainly take a while to fix all of these problems and add the requested features. I think I'm going to try to fix the environment variable problem first, so that I can release a beta that can at least launch T2TP properly. This first beta will take a few weeks (but the priority may be a problem and delay the release). Then I'll slowly implement the new features (possibly releasing a small beta featuring the 'output folder/image prefix' override thing), eventually releasing a near-final beta which should have most of the features suggested here as well as most of the features I removed when changing the program so it would work with T2TP rather than TG 0.9. After all of that, I will look at dividing frames.

Finally, I have a question for Oshyan: is the command line structure due to change significantly between now and the proper release of TG2?

bigben

Quote from: Dark Fire on March 22, 2007, 05:09:51 PM

I considered adding the ability to move all the files needed into a single folder to software I wrote for TG 0.9, but I eventually decided to drop the idea. T2TP is forcing me to consider that option again. Stripping out the file names shouldn't be too hard, but it would take quite a while to write because I would have to be absolutely sure that no other data is stripped out.

I've been thinking about what direction my database tools should take (generate frame specific TGDs or generate animation settings files to embed in the TGD) and I'm still undecided. One of the things I was looking at was parsing the TGD file to extract each of the variables with cross checking of output field values (text/numeric, number of values etc...) against an xml specification... from this.... the process of locating external file information could be simplified by only checking those tags that contain filenames and excluding node names which might also contain filenames (e.g. the XFrog tree models). This would reduce the chance of affecting other data.

I should be able to add this functionality to my database if nothing else is available... which would save me some time and disk space.

Using your software is a step further along the approach I have taken with my own crude render farms. I'd generate a .bat file for each computer and then manually start it (via VNC to save walking around the building). The main reason for this was that with long animations I ran progressive renders, rendering every 64 frames, then every 32,16,8,4,2,1 frames.  I could still do this using you software with a simple modification of my database to generate a single joblist with a job for each frame. This would be vastly more efficient as it won't matter that there's a mix of PC specifications in the farm  (and believe me, they go from race horses to cows ;)) or one or more get turned off along the way.

I'll look forward to the next testing round :) Drop me a line off list if there's anything else you want me to test along the way.

Oshyan

Dark Fire, I don't expect the commandline structure to change significantly, but Matt would know more than I.

- Oshyan

Dark Fire

Quote from: Oshyan on March 22, 2007, 10:21:09 PM
Dark Fire, I don't expect the commandline structure to change significantly, but Matt would know more than I.

- Oshyan
I can deal with insignificant changes as long as somebody tells me about them. Thanks Oshyan.

Quote from: bigben on March 22, 2007, 07:36:13 PM
I've been thinking about what direction my database tools should take (generate frame specific TGDs or generate animation settings files to embed in the TGD) and I'm still undecided. One of the things I was looking at was parsing the TGD file to extract each of the variables with cross checking of output field values (text/numeric, number of values etc...) against an xml specification... from this.... the process of locating external file information could be simplified by only checking those tags that contain filenames and excluding node names which might also contain filenames (e.g. the XFrog tree models). This would reduce the chance of affecting other data.
I really need to have a good look at the TGD file format. One problem that I have realised since yesterday is that the format may still change, which may cause problems later.

Quote from: bigben on March 22, 2007, 07:36:13 PM
Using your software is a step further along the approach I have taken with my own crude render farms.
Hehe. My software is actually quite crude too.

Quote from: bigben on March 22, 2007, 07:36:13 PM
I'd generate a .bat file for each computer and then manually start it (via VNC to save walking around the building). The main reason for this was that with long animations I ran progressive renders, rendering every 64 frames, then every 32,16,8,4,2,1 frames.  I could still do this using you software with a simple modification of my database to generate a single joblist with a job for each frame. This would be vastly more efficient as it won't matter that there's a mix of PC specifications in the farm  (and believe me, they go from race horses to cows ;)) or one or more get turned off along the way.
You may have noticed that the job list file is actually an '.ini' file in disguise. It appears that you are trying to exploit two of the great features my software has - the ability to keep fast computers supplied with work to do, and the ability to work out when a computer has been turned off, crashed or been disconnected from the network.

Quote from: bigben on March 22, 2007, 07:36:13 PM
I'll look forward to the next testing round :) Drop me a line off list if there's anything else you want me to test along the way.
OK. Thanks.

ANNOUNCEMENT:
This is an announcement that I've been trying to put off for a long time, but I'm afraid I can't put it off any longer. In order to actually get some work done on my software and also to get work done for my exams this summer (and beyond), I will have to almost completely stop using for forums. I will keep an eye on general developments and topics like this, which I seem to have become an intrinsic part of, as well as the two topics I started (yes, only two - despite being the third highest poster). This post will put me on 567 posts - 67 posts ahead of old_blaggard (the closest person behind me), 364 posts behind Oshyan and 395 posts behind Will (who both seem to live on these forums). I will undoubtedly drift down the rankings now and people will probably start to forget me as I will only post very occasionally, but I am glad that I have been a significant early force on these forums. So, in conclusion, goodbye and good luck everyone! (Especially the Planetside team).

While I still have a decent opportunity, I am going to pour a final list of suggestions onto these forums:
1. Terragen needs a proper logo.
2. Once Terragen has a proper logo, merchandise should be sold (I can't afford T2TP, but I will be able to buy merchandise)!
3. The eternal suggestion: make it render faster!
4. Make the icons in TG2 reflect the quality of the images that can be produced with TG2 - at the moment, the icons look very old...
5. More features in the free version. I think TG 0.9 had a good balance - reasonably large images, animation features and a working command line. It seems strange that as screens get bigger and better, the free version of T2TP won't let you render images bigger than 800x600.

I could put down more, but the five suggestions above are probably my top five.

old_blaggard

We'll miss you, DarkFire.  Hopefully you'll be able to pop back in after exams are done.  I've got exams during May myself, but they're still far enough away that I can pretend they aren't there :P.  Maybe by the time you get back, everything on your wish list will exist ;).
http://www.terragen.org - A great Terragen resource with models, contests, galleries, and forums.

Dark Fire

#36
Quote from: old_blaggard on March 23, 2007, 04:38:32 PM
I've got exams during May myself, but they're still far enough away that I can pretend they aren't there.
I could still pretend my exams are not there, but I am currently trying to beat the rest of my maths class by getting the most maths exams in a row with a score of 100%. I currently have the joint record (four consecutive exams), but I am the only person currently still on a 100% streak. That means that if I get 100% in May, I alone will hold the record :D. I need to work hard to be sure of getting 100% in May, so that's a major reason for me leaving for now...and there's the small fact that I still have a quarter of my Music Technology Coursework to do before May. :'(

Quote from: old_blaggard on March 23, 2007, 04:38:32 PM
Maybe by the time you get back, everything on your wish list will exist ;).
Suggestions 1 to 4 are possible, but I can't see suggestion 5 happening - TG2 seems to be a much more serious product than TG 0.9...

Adam Chrystie

I use The Condor Project to handle my renderfarm needs.

It works with Windows,Os X,Linux all on the same pool.

I got it working with Terragen 2 OS X commmand line.

--------------------
Adam Chrystie
3D Artist & Infrastructure/Pipeline Developer

Dark Fire

Indeed, there is already lots of software out there that can help to manage render farms, but software written specifically for Terragen will always be more useful because it will be able to carry out tasks that are specific to TG2. Software written specifically for Terragen will also be a lot quicker and easier to use. People who already use render farm management software should probably stick to what they're using, but people who are new to render farms would probably find it easier to learn how to use software written specifically for Terragen.

Dark Fire

Progress report:
I think I've now got the client to launch tgdcli.exe correctly, with the priority specified by the user. Could people with copies of T2TP with working command line interfaces please test this software.

An extra program called 'Diagnostic Mode Client.exe', which shows lots of information that will be useful if things go wrong, is included.

It should be noted that even the normal 'Client.exe' is nowhere near finished - the annoying command prompt box, for example, will be hidden when I release version 1.0. Details shown in the box will hopefully be fed through onto the Client interface in version 1.0.

Dark Fire

6 downloads of the new version already. Could somebody please tell me if T2TP is launching, opening projects and rendering images correctly?

old_blaggard

I'm sorry, but I'm on a Mac, and I can't run .exe's.  Good to see you back, by the way.
http://www.terragen.org - A great Terragen resource with models, contests, galleries, and forums.

Dark Fire

I'm only temporarily back - I should really be finishing coursework now...

Dark Fire

I have added another program to the package: 'Client (Method 2).exe'. This new program tries to set the environment variable using a less messy method than 'Client.exe' and 'Diagnostic Mode Client.exe'.

Here is a summary of the feedback I need:
1. Does 'Client.exe' run T2TP properly?
2. If 'Client.exe' does not run T2TP properly, what does the command prompt window display when 'Diagnostic Mode Client.exe' tries to run T2TP?
3. Does 'Client (Method 2).exe' run T2TP properly?

A copy of T2TP with a working command line will be needed for these tests (which is why I am not performing them myself). T2TP should be successfully run by each program, but what I am looking for is whether or not T2TP opens the correct project and renders the desired frame.

Please leave feedback in a reply to this topic, or on my website (on which guests can post).

tomatomic

well, i dont completely understand, but i tried running it per instructions, and it made a bunch of bat files that seem to call up the render client without actually rendering frames.. while the monitor or the client was reporting actually rendering the frames.
note: getting error codes 3O and 4R

i dont have a lot of time to troubleshoot this, but if it turns out to be a quick solution, id be happy to spend a little time with feedback to save me the trouble of chasing individual frames.. my farm has different speeds of processors, and my sequence has frames that will take varying times to render.. so this could really make things easier for me.

it pm you on the forums with my email