backburner for Distributed rendering with terragen2

Started by miqtidar, November 25, 2010, 07:51:44 PM

Previous topic - Next topic

stevesideas

#15
Apologies. Two more questions. Do I need to specify an output path because I see these frames being rendered but when I check the path that was specified in terragen, the files are not there ? Will the batch render automatically use the Full render node as oppose to the quick render node ? or do I need to delete the quick render node ?

UPDATED: I ended up mapping a network drive to a letter so that it is easier to work with the nodes. The problem I have now is that using the script the render nodes keep overwriting the same output file.

Matt

Quote from: stevesideas on June 28, 2013, 02:01:12 PM
The problem I have now is that using the script the render nodes keep overwriting the same output file.

If you put the string "%04d" (without quotes) somewhere in the filename, that will be replaced by the frame number. As long as Backburner is telling Terragen which frame to render, that should work. If Terragen is rendering the same frame over and over, it means the command line isn't right.

Matt
Just because milk is white doesn't mean that clouds are made of milk.

stevesideas

#17
Quote from: Matt on June 28, 2013, 09:52:05 PM
Quote from: stevesideas on June 28, 2013, 02:01:12 PM
The problem I have now is that using the script the render nodes keep overwriting the same output file.

If you put the string "%04d" (without quotes) somewhere in the filename, that will be replaced by the frame number. As long as Backburner is telling Terragen which frame to render, that should work. If Terragen is rendering the same frame over and over, it means the command line isn't right.

Matt

I'll show you my setup:

This is the render node inside Terragen2 (looks fine to me:

[attachimg=1]

and this is the command line inside a .BAT file:


cmdjob -jobname:"terragentest1" -manager:192.168.0.4 -tp_start:1 -numTasks:250 -taskname:1 "c:\Program Files\Planetside Software\Terragen 2 Deep Edition\tgdcli.exe" "H:\terragen_render_files\landscape_.tgd" -hide -exit -r -f %tp1


I'm not sure what the problem is. Do you mind taking a look at this ?

stevesideas

I did a bit of searching and it seems that when you run the code from a .BAT file you need to change the syntax to: %%tp1 instead of just %tp1. This seems to have resolved the issue and now all frames are being rendered.  Thanks very much for all your help.

Matt

Glad to hear that works :) Thanks for the update.

Matt
Just because milk is white doesn't mean that clouds are made of milk.

stevesideas

Has this changed for TG3 ? I can't seem to get rendering via backburner to work any more.

stevesideas

Silly me. I need to put the license file inside the node folders.

FlynnAD

Hi Steve,

Have you setup your Backburner command script to properly save the output files?

1) For me, it seems that TG3 is saving the output files NOT where it's listed in the Render Node's Output slot, but in the general TG3 Preferences dialog box\ File Saving\ Customize Default Paths\ Item: Sequence Output\ Path. And this is strange because I have to set this for each render node; Backburner doesn't do it globally.

2) When using Backburner, the renderings are getting saved as .bmps, even if I specify something else in EITHER the Render Node Output slot or in the general TG3 Preferences dialog box\ File Saving\ Render save format: (such as tif or exr). When I try rendering this out of the command line straight to TG3 (not through Backburner), the filetype saves correctly, plus the output images too. However, I don't want to use the pure command line without Backburner because it's a hassle to not be able to control on/off rendering machines.

3) Were you able to get only the rendering box to show up (your earlier question; like 3dsmax does)?

Thanks,
Matt

stevesideas

Quote from: FlynnAD on September 07, 2013, 07:26:46 PM
Hi Steve,

Have you setup your Backburner command script to properly save the output files?

1) For me, it seems that TG3 is saving the output files NOT where it's listed in the Render Node's Output slot, but in the general TG3 Preferences dialog box\ File Saving\ Customize Default Paths\ Item: Sequence Output\ Path. And this is strange because I have to set this for each render node; Backburner doesn't do it globally.

2) When using Backburner, the renderings are getting saved as .bmps, even if I specify something else in EITHER the Render Node Output slot or in the general TG3 Preferences dialog box\ File Saving\ Render save format: (such as tif or exr). When I try rendering this out of the command line straight to TG3 (not through Backburner), the filetype saves correctly, plus the output images too. However, I don't want to use the pure command line without Backburner because it's a hassle to not be able to control on/off rendering machines.

3) Were you able to get only the rendering box to show up (your earlier question; like 3dsmax does)?

Thanks,
Matt


This is the latest batch file I set up:

cmdjob -jobname:"test_render" -manager:192.168.0.4 -tp_start:1 -numTasks:1000 -taskname:1 "tgdcli" "T:\render_files\test.tgd" -hide -exit -r -f %%tp1

I made sure that my system variables were set up correctly for each node and my main workstation so that I could just have "tgdcli" in the bat file script.

I then created a folder and set this as a network drive with the same letter on each node and the main workstation. ("T" in this case").

After this I create a folder on the T drive called "render_files" and another called "render_output".

In the render node I set the output to this render path "render_output" making sure that it is a network path.

This method should output all your frames to that folder from each node you have connected.

I normally specific a name in the render output path (inside the render node in TG3) to be something like "scene_.%04d.exr". Not sure why you would be having problems saving out to the file type specified.

When you say rendering box I assume you are talking about render buffer as displayed when something is rendering. No, I could not work out how to do this.

I hope this helps you :)

FlynnAD

#24
Thanks Steve.

Finally got it to work after many tries. Your response was the key.

What I found out was that not only inside the actual TG3 file does the Render Node's Output path need to be a network drive, but even the batch command line needs to have a network drive.

For example, my problem stemmed from the fact that all my slave machines have a D:\_temp3ds folder where I save each machine's output locally. Later I move and consolidate everything. Yet doing this (using D:\ like your T:\) had TG3 saving each file as a .bmp only. However, changing only the .tgd's Render Node Output Path to a network drive did not solve the problem. Each slave machine still dumped each rendered frame, without any rendered elements, as a bmp only, into the folder in TG3's\Preferences\File Saving\Sequenced Output location.

So the batch command also must have the location of the .tgd file as a network drive. I still wanted to use the D:\ drive on my main machine, so I went through the network and called it "\\Flynn01\D-Bern\filelocation".

FYI to anyone else, the "-x" variable noted on the first page of this post does not seem to be needed anymore.

cmdjob -jobname:"Backburner-job-nickname" -manager:10.0.0.2 -priority:50 -taskList:"D:\_tempTG3\gathered_files\thisisthetasklist.txt" -taskName:2 "C:\Program Files\Planetside Software\Terragen 3\tgdcli.exe" "\\FLYNN01\D-Bern\_tempTG3\gathered_files\defaultTG3-networksave.tgd" -hide -exit -r -f %%tp1

Also, for anyone reading this at a later date:

I was trying NOT to hard-set the TERRAGEN_PATH environmental variable in case I used TG2.5 again, per the recommendation in TG3 Doc's file. However, I ended up hard setting it on all the nodes anyways, so I cannot say if it did help in the end or not. For anyone who wants a quick way of setting it for all their render nodes, simply copy the attached "TG3 environ...txt" file, change the extension to .bat, and run it on each machine. Takes two seconds.

Other notes:

The Backburner help file says that the -manager:IP.IP.IP.IP address is not needed and it can use the default port, but I found the TG3 file loads into Backburner faster if you do include this tag.

You can set the -priority:50 or whatever you want, Critical, 01, etc.

In TG3's Docs\win_command_line.txt file, it notes how you can set up specific frames within a batch command (non-Backburner), but Backburner does not pay attention to that notation. To do so in Backburner, use a Tasklist. One is attached for starters. Use the Excel to quickly copy and fill the frames you want, even odd or spaced frames. Then copy all the data from those two columns into a txt file. This way the job in Backburner will show up with the same frame # as the actual frame you are rendering. Without the tasklist, you may be rendering frames 14-17, but it will be listed in Backburner as tasks #1-4. That makes it hard to track if a specific frame is having a problem. The tasklist can be locally referenced in the .bat file.

The tasklist attached is the equivalent to specifying: -f 1-3,26-41/5,90-93.

Cheers,
Matt

FlynnAD

Hi Planetside,

Is there a way to show the UI (or more ideally, just show the rendered image frame buffer)?

On the command prompt lines in the discussions above, we're adding the notation -hide and -exit. I thought the documentation for TG3 somewhere said that the -hide and -exit were now the default for command prompt rendering, so this notation isn't even needed anymore. Is there a command to SHOW the rendered frame buffer? This would allows users to stop incorrect renderings quickly without letting the computer render for hours in the dark.

Thanks,
Matt

Matt

Hi Matt,

There isn't a way to do this yet, but I'm thinking about it. It would be straightforward for us to add a "-show" argument to the command line that would show the whole GUI. This would be simple with the least likelihood of introducing bugs and requiring extra testing. How important is it to you to limit it to just the Render View? I guess we'd want to allow the Errors and Warnings window to be shown as well as the Render View?

Matt
Just because milk is white doesn't mean that clouds are made of milk.

FlynnAD

Quote from: Matt on September 17, 2013, 11:29:20 PM
How important is it to you to limit it to just the Render View? I guess we'd want to allow the Errors and Warnings window to be shown as well as the Render View?

Hi Matt,

First off, this starts a new wish list because Planetside added all the functionality in TG3 that I was previously hoping for, so thank you.

For whatever part of the GUI you can make visible, my hope would be at least the Render window, because then you could see visually 1) if there were errors, and 2) get a rough sense of where the process is, so you know if the image is almost finished rendering or only in the middle.

I figured showing the entire GUI would take up more CPU use than hiding it, so really, all that's requested is the Render Window. I personally would prefer slave machines to render as fast as they can, but honestly, showing the full GUI is a better tradeoff than the lost time rendering an error-ed frame because there is no GUI. Sure, errors and warnings would be helpful, if not a headache to include.

Thanks,
Matt

The3DStig

#28
I've got a TG3 temporary license and I'm really enjoying it!  I've even got the higher-ups looking into purchasing which should be going through in a couple of weeks!

I'd really like to get my small render farm in on the action using backburner since I already use it to manage my Vray/Max work.  I've been through this forum several times and I still can't get this silly thing to work!  I know it's possible... in a fleeting moment of madness I just starting chopping away at the code and SUDDENLY(!) Terragen opened!  ... but then nothing happened.

Anyway, if someone wouldn't mind having a look at this and pointing me in the right direction I'd be very grateful!

If I can get it working on my workstation (the only one with a license for now) I'm sure I can work out the others once I have my full licenses.
I'm using the version of backburner that requires IPv6... does that make a difference?

Here's my cmd line code:
cmdjob -jobname:"Backburner-job-nickname" -manager:2002:c01b:5753::c01b:5753 -priority:51 -taskList:"D:\Current Projects\TerragenTests\TaskList_Maker1\TaskList Maker-120.txt" -taskName:2 "C:\Program Files\Planetside Software\Terragen 3\tgd.exe" "D:\1100635\Documents\Terragen Files\GettingStarted\GettingStarted-022.tgd" -hide -exit -r -f %%tp1

I keep getting a not too helpful error message.  I'm including a capture of my cmd line window.
Thanks in advance!

The3DStig

#29
Progress!  ... well sort of.

Using this code I don't get any errors but nothing happens... I let it sit for about 15 minutes.  The cmd line just says "the job will trigger 120 task" and then nothing happens.
Anyway, here's what got me this far:

cmdjob -jobname:"Backburner-job-nickname" -priority:50 -manager:2002:c01b:5753::c01b:5753 -port:3234 -servers:ZRCRA062799 -taskList:"D:\Current Projects\TerragenTests\TaskList_Maker1\TaskList Maker-120.txt" -taskName:2 "C:\Program Files\Planetside Software\Terragen 3\tgdcli.exe" "D:\1100635\Documents\Terragen Files\GettingStarted\GettingStarted-022.tgd" -r -f %%tp1