Planetside Software Forums

Hello every one.

Can somebody explain how i can use backburner for Distributed rendering with terragen2?

I mean using 2 computer together to render one single image or may be 3 computers?

I have looked to this post but process is not explained.

http://forums.planetside.co.uk/index.php?topic=5421.0

nobody is there to answer my question

Unfortunately I don't have more than 1 PC to do something similar like you :(

My guess is that the guy in the link you posted is splitting his tgd's up in render-regions to be rendered.
These separate tgd's are then distributed via the rendermanager.

As far as I know, there's no other way.

Martin

Sorry about the late response. For splitting up a single render, you must use Crop Regions (ideally with some overlap) and create multiple TGDs, one with each crop region, which are then distributed for rendering across several computers. The distributed rendering would be controlled by Backburner using its commandline support by calling the Terragen executable with a different TGD file on each rendering machine. Once each crop area is rendered you then need to reassemble them using a stitching application of some kind. Photoshop can handle this, as can Panotools/Huigin (free), and many other systems.

This is not quite as simple as a one-click operation, it requires a little bit of setup, so it's not something you would likely use on a daily basis. But if you are making a very high resolution or high quality final image it can be useful to distribute the load across several systems to reduce render time.

- Oshyan

Thanks Oshyan,
That was the great answer i wanted to know it helped alot.

By the way Thanks for the new update.

For help on submitting your actual files to backburner using the commandline (as long as your environment variables are set correctly), you can use the following line and modify it to suit your needs:

exec cmdjob -jobname:$jobName -manager:$managerAddress -priority:$priority -description:$description -tp_start:$firstFrame -numTasks:$numTasks -taskname:1 $tgdExe -x $file %tp1

Where:
$jobName = The name of your job.
$managerAddress = The IP address of your manager, most probably something similar to 192.168.10.1.
$priority = The priority value of your job.
$firstFrame = The first frame of your animation, typically "1".
$numTasks = The number of frames in your animation. If it's a single frame, it will be "1".
$tgdExe = The path to the Terragen executable.
$file = The path and name of the file to render, eg: "S:\Terragen\file.tgd".

An example:
exec cmdjob -jobname:"Sunset Field" -manager:192.168.10.1 -priority:50 -description:"crop1" -tp_start:1 -numTasks:1 -taskname:1 "D:\Program Files (32 bit)\Planetside Software\Terragen 2 Deep Edition\tgd.exe" -x "S:\Terragen\file.tgd" %tp1

thanks njen,

Hi all,

This is an old thread, but still seems to be the most relevant. Oshyan referenced it just a couple months ago: (topic 15739).

I am using Backburner v.2012 and TG2.5.5.0. Using njen's code, I revised it to:

cmdjob -jobname:"Sunset Field" -tp_start:1 -taskname:1 "C:\Program Files\Planetside Software\Terragen 2 Deep Edition\tgd.exe" "D:\file.tgd" %tp1.

The portions of the command for the $manageraddress, $priority, $description are not required if you're sending it to the default Backburner manager. It still works fine.

I also got an error when using "-x" before the .tgd file; Terragen 2 said that it couldn't open the file. So I took this off, and there were no errors opening the file.

Now the file loaded into Backburner's queue properly, TG2 booted up, then opened the file, but then did not do anything further. Nothing rendered. Are there more commands needed in cmdjob to run the renderer?

Also, if you have numerous Render nodes in your scene (Full Render, Quick Render, Animation Fast Test Render, etc), does this cmdjob code by default use the checked Master Render node?

If you want to render an animation (not through Backburner), you have to click on the Render node, then the Sequence/Output tab, then on Render Sequence. Just hitting "Render Image" will never save the output; the user has to do that himself. How can you make sure Backburner saves the file and doesn't just render without saving?

Lastly, in a recent TG3 post, Oshyan wrote that "Node rendering on Mac/Win is done through the existing CLI version. You can use a commandline flag to hide the UI if desired.". What is the CLI version of node rendering? Is it easier than Backburner?

Attached is a list of the Backburner cmdjob commands and a test tgd file.

Thanks for the help,
-Matt

Hi Matt,

Following the name of the Terragen executable and project file, you need to pass additional switches and parameters to tell Terragen to render a frame (or set of frames). Here are some examples taken from the win_command_line.txt document in the docs folder of your installation:

Quote
Run Terragen without the GUI, render frame 42, then close:

    "%TERRAGEN_PATH%/tgdcli" -p myproject.tgd -hide -exit -r -f 42

Run Terragen without the GUI, render frames 1,4,51,53,55,57,59, then close:

    "%TERRAGEN_PATH%/tgdcli" -p myproject.tgd -hide -exit -r -f 1,4,51-60/2

What you should take away from the above is the use of "-hide -exit -r -f" followed by the frame (or set of frames) that you want to render. Without these, Terragen will simply load the project and sit there waiting for you to do something.

You should probably also use tgdcli instead of tgd (on Windows). The differences are minimal, but I think tgdcli.exe handles returning of control to the calling process in a different way to tgd.exe.

For more information and other options available on the command line, refer to the docs folder of your installation.

Matt

Quote from: Matt on June 15, 2013, 06:43:21 AM
<...>
For more information and other options available on the command line, refer to the docs folder of your installation.

Matt,

Thank you for the script and location of the docs folder. Really helpful.

-Matt

This thread has really helped me but I need some additional help if you don't mind.

I have 4 render nodes each with Terragen 2 installed.

I have this .bat file setup:

cmdjob -jobname:"terragentest1" -manager:192.168.0.4 -tp_start:1 -taskname:1 "c:\Program Files\Planetside Software\Terragen 2 Deep Edition\tgdcli.exe" "\\MAINCOMP\graphics files\MAXSCENES\aircraft_test\terragen_files\landscape_.tgd" -exit -r -f 1-250

This seems to work to an extent but only sends one task (which I assume is the full 1-250 frames) to one render node. How do I split the animation between the nodes and render a sequence where backburner allocates a frame depending on the node availability? Similar to the way 3dsmax renders are handled.

Any help appreciated

I am not familiar enough with Backburner to answer the question. But I can give some pointers. Backburner will somehow need to be told the frame range in its own syntax. I don't know what that syntax is, but it doesn't appear to be in your example. The "1-250" in your example is probably just being sent directly to Terragen. That might need to be replaced by some Backburner variable so that Backburner tells Terragen a different frame each time it calls Terragen.

Matt

From what I can piece together from the above information, I think you might need to add -numTasks:250 after -tp_start:1 and then add -f %tp1 at the very end of the Terragen part. I can't test it from here though.

So, if I modify your code it would be this:

cmdjob -jobname:"terragentest1" -manager:192.168.0.4 -tp_start:1 -numTasks:250 -taskname:1 "c:\Program Files\Planetside Software\Terragen 2 Deep Edition\tgdcli.exe" "\\MAINCOMP\graphics files\MAXSCENES\aircraft_test\terragen_files\landscape_.tgd" -exit -r -f %tp1

I'm really not sure though.

Matt

Thanks for this. I'll let you know how things go.

Quote from: Matt on June 28, 2013, 03:32:24 AM
From what I can piece together from the above information, I think you might need to add -numTasks:250 after -tp_start:1 and then add -f %tp1 at the very end of the Terragen part. I can't test it from here though.

So, if I modify your code it would be this:

cmdjob -jobname:"terragentest1" -manager:192.168.0.4 -tp_start:1 -numTasks:250 -taskname:1 "c:\Program Files\Planetside Software\Terragen 2 Deep Edition\tgdcli.exe" "\\MAINCOMP\graphics files\MAXSCENES\aircraft_test\terragen_files\landscape_.tgd" -exit -r -f %tp1

I'm really not sure though.

Matt

Brilliant ! This is working a treat. Thank you very much !

Just a couple more questions if possible. Is it possible to 1. Just show the render window on each render node so I can see the progression per frame ? and 2. I have added the path to terragen in the environment variables but when I set the terragen path as "tgdcli.exe" in the BAT file there is an error when the nodes try to batch render. They can't see terragen path for some reason. I have to set it to the full path for it to work.

Apologies. Two more questions. Do I need to specify an output path because I see these frames being rendered but when I check the path that was specified in terragen, the files are not there ? Will the batch render automatically use the Full render node as oppose to the quick render node ? or do I need to delete the quick render node ?

UPDATED: I ended up mapping a network drive to a letter so that it is easier to work with the nodes. The problem I have now is that using the script the render nodes keep overwriting the same output file.

Quote from: stevesideas on June 28, 2013, 02:01:12 PM
The problem I have now is that using the script the render nodes keep overwriting the same output file.

If you put the string "%04d" (without quotes) somewhere in the filename, that will be replaced by the frame number. As long as Backburner is telling Terragen which frame to render, that should work. If Terragen is rendering the same frame over and over, it means the command line isn't right.

Matt

Quote from: Matt on June 28, 2013, 09:52:05 PM

Quote from: stevesideas on June 28, 2013, 02:01:12 PM
The problem I have now is that using the script the render nodes keep overwriting the same output file.

If you put the string "%04d" (without quotes) somewhere in the filename, that will be replaced by the frame number. As long as Backburner is telling Terragen which frame to render, that should work. If Terragen is rendering the same frame over and over, it means the command line isn't right.

Matt

I'll show you my setup:

This is the render node inside Terragen2 (looks fine to me:

[attachimg=1]

and this is the command line inside a .BAT file:


cmdjob -jobname:"terragentest1" -manager:192.168.0.4 -tp_start:1 -numTasks:250 -taskname:1 "c:\Program Files\Planetside Software\Terragen 2 Deep Edition\tgdcli.exe" "H:\terragen_render_files\landscape_.tgd" -hide -exit -r -f %tp1

I'm not sure what the problem is. Do you mind taking a look at this ?

I did a bit of searching and it seems that when you run the code from a .BAT file you need to change the syntax to: %%tp1 instead of just %tp1. This seems to have resolved the issue and now all frames are being rendered.  Thanks very much for all your help.

Glad to hear that works :) Thanks for the update.

Matt

Has this changed for TG3 ? I can't seem to get rendering via backburner to work any more.

Silly me. I need to put the license file inside the node folders.

Hi Steve,

Have you setup your Backburner command script to properly save the output files?

1) For me, it seems that TG3 is saving the output files NOT where it's listed in the Render Node's Output slot, but in the general TG3 Preferences dialog box\ File Saving\ Customize Default Paths\ Item: Sequence Output\ Path. And this is strange because I have to set this for each render node; Backburner doesn't do it globally.

2) When using Backburner, the renderings are getting saved as .bmps, even if I specify something else in EITHER the Render Node Output slot or in the general TG3 Preferences dialog box\ File Saving\ Render save format: (such as tif or exr). When I try rendering this out of the command line straight to TG3 (not through Backburner), the filetype saves correctly, plus the output images too. However, I don't want to use the pure command line without Backburner because it's a hassle to not be able to control on/off rendering machines.

3) Were you able to get only the rendering box to show up (your earlier question; like 3dsmax does)?

Thanks,
Matt

Quote from: FlynnAD on September 07, 2013, 07:26:46 PM
Hi Steve,

Have you setup your Backburner command script to properly save the output files?

1) For me, it seems that TG3 is saving the output files NOT where it's listed in the Render Node's Output slot, but in the general TG3 Preferences dialog box\ File Saving\ Customize Default Paths\ Item: Sequence Output\ Path. And this is strange because I have to set this for each render node; Backburner doesn't do it globally.

2) When using Backburner, the renderings are getting saved as .bmps, even if I specify something else in EITHER the Render Node Output slot or in the general TG3 Preferences dialog box\ File Saving\ Render save format: (such as tif or exr). When I try rendering this out of the command line straight to TG3 (not through Backburner), the filetype saves correctly, plus the output images too. However, I don't want to use the pure command line without Backburner because it's a hassle to not be able to control on/off rendering machines.

3) Were you able to get only the rendering box to show up (your earlier question; like 3dsmax does)?

Thanks,
Matt

This is the latest batch file I set up:

cmdjob -jobname:"test_render" -manager:192.168.0.4 -tp_start:1 -numTasks:1000 -taskname:1 "tgdcli" "T:\render_files\test.tgd" -hide -exit -r -f %%tp1

I made sure that my system variables were set up correctly for each node and my main workstation so that I could just have "tgdcli" in the bat file script.

I then created a folder and set this as a network drive with the same letter on each node and the main workstation. ("T" in this case").

After this I create a folder on the T drive called "render_files" and another called "render_output".

In the render node I set the output to this render path "render_output" making sure that it is a network path.

This method should output all your frames to that folder from each node you have connected.

I normally specific a name in the render output path (inside the render node in TG3) to be something like "scene_.%04d.exr". Not sure why you would be having problems saving out to the file type specified.

When you say rendering box I assume you are talking about render buffer as displayed when something is rendering. No, I could not work out how to do this.

I hope this helps you :)

Thanks Steve.

Finally got it to work after many tries. Your response was the key.

What I found out was that not only inside the actual TG3 file does the Render Node's Output path need to be a network drive, but even the batch command line needs to have a network drive.

For example, my problem stemmed from the fact that all my slave machines have a D:\_temp3ds folder where I save each machine's output locally. Later I move and consolidate everything. Yet doing this (using D:\ like your T:\) had TG3 saving each file as a .bmp only. However, changing only the .tgd's Render Node Output Path to a network drive did not solve the problem. Each slave machine still dumped each rendered frame, without any rendered elements, as a bmp only, into the folder in TG3's\Preferences\File Saving\Sequenced Output location.

So the batch command also must have the location of the .tgd file as a network drive. I still wanted to use the D:\ drive on my main machine, so I went through the network and called it "\\Flynn01\D-Bern\filelocation".

FYI to anyone else, the "-x" variable noted on the first page of this post does not seem to be needed anymore.

cmdjob -jobname:"Backburner-job-nickname" -manager:10.0.0.2 -priority:50 -taskList:"D:\_tempTG3\gathered_files\thisisthetasklist.txt" -taskName:2 "C:\Program Files\Planetside Software\Terragen 3\tgdcli.exe" "\\FLYNN01\D-Bern\_tempTG3\gathered_files\defaultTG3-networksave.tgd" -hide -exit -r -f %%tp1

Also, for anyone reading this at a later date:

I was trying NOT to hard-set the TERRAGEN_PATH environmental variable in case I used TG2.5 again, per the recommendation in TG3 Doc's file. However, I ended up hard setting it on all the nodes anyways, so I cannot say if it did help in the end or not. For anyone who wants a quick way of setting it for all their render nodes, simply copy the attached "TG3 environ...txt" file, change the extension to .bat, and run it on each machine. Takes two seconds.

Other notes:

The Backburner help file says that the -manager:IP.IP.IP.IP address is not needed and it can use the default port, but I found the TG3 file loads into Backburner faster if you do include this tag.

You can set the -priority:50 or whatever you want, Critical, 01, etc.

In TG3's Docs\win_command_line.txt file, it notes how you can set up specific frames within a batch command (non-Backburner), but Backburner does not pay attention to that notation. To do so in Backburner, use a Tasklist. One is attached for starters. Use the Excel to quickly copy and fill the frames you want, even odd or spaced frames. Then copy all the data from those two columns into a txt file. This way the job in Backburner will show up with the same frame # as the actual frame you are rendering. Without the tasklist, you may be rendering frames 14-17, but it will be listed in Backburner as tasks #1-4. That makes it hard to track if a specific frame is having a problem. The tasklist can be locally referenced in the .bat file.

The tasklist attached is the equivalent to specifying: -f 1-3,26-41/5,90-93.

Cheers,
Matt

Hi Planetside,

Is there a way to show the UI (or more ideally, just show the rendered image frame buffer)?

On the command prompt lines in the discussions above, we're adding the notation -hide and -exit. I thought the documentation for TG3 somewhere said that the -hide and -exit were now the default for command prompt rendering, so this notation isn't even needed anymore. Is there a command to SHOW the rendered frame buffer? This would allows users to stop incorrect renderings quickly without letting the computer render for hours in the dark.

Thanks,
Matt

Hi Matt,

There isn't a way to do this yet, but I'm thinking about it. It would be straightforward for us to add a "-show" argument to the command line that would show the whole GUI. This would be simple with the least likelihood of introducing bugs and requiring extra testing. How important is it to you to limit it to just the Render View? I guess we'd want to allow the Errors and Warnings window to be shown as well as the Render View?

Matt

Quote from: Matt on September 17, 2013, 11:29:20 PM
How important is it to you to limit it to just the Render View? I guess we'd want to allow the Errors and Warnings window to be shown as well as the Render View?

Hi Matt,

First off, this starts a new wish list because Planetside added all the functionality in TG3 that I was previously hoping for, so thank you.

For whatever part of the GUI you can make visible, my hope would be at least the Render window, because then you could see visually 1) if there were errors, and 2) get a rough sense of where the process is, so you know if the image is almost finished rendering or only in the middle.

I figured showing the entire GUI would take up more CPU use than hiding it, so really, all that's requested is the Render Window. I personally would prefer slave machines to render as fast as they can, but honestly, showing the full GUI is a better tradeoff than the lost time rendering an error-ed frame because there is no GUI. Sure, errors and warnings would be helpful, if not a headache to include.

Thanks,
Matt

I've got a TG3 temporary license and I'm really enjoying it!  I've even got the higher-ups looking into purchasing which should be going through in a couple of weeks!

I'd really like to get my small render farm in on the action using backburner since I already use it to manage my Vray/Max work.  I've been through this forum several times and I still can't get this silly thing to work!  I know it's possible... in a fleeting moment of madness I just starting chopping away at the code and SUDDENLY(!) Terragen opened!  ... but then nothing happened.

Anyway, if someone wouldn't mind having a look at this and pointing me in the right direction I'd be very grateful!

If I can get it working on my workstation (the only one with a license for now) I'm sure I can work out the others once I have my full licenses.
I'm using the version of backburner that requires IPv6... does that make a difference?

Here's my cmd line code:
cmdjob -jobname:"Backburner-job-nickname" -manager:2002:c01b:5753::c01b:5753 -priority:51 -taskList:"D:\Current Projects\TerragenTests\TaskList_Maker1\TaskList Maker-120.txt" -taskName:2 "C:\Program Files\Planetside Software\Terragen 3\tgd.exe" "D:\1100635\Documents\Terragen Files\GettingStarted\GettingStarted-022.tgd" -hide -exit -r -f %%tp1

I keep getting a not too helpful error message.  I'm including a capture of my cmd line window.
Thanks in advance!

Progress!  ... well sort of.

Using this code I don't get any errors but nothing happens... I let it sit for about 15 minutes.  The cmd line just says "the job will trigger 120 task" and then nothing happens.
Anyway, here's what got me this far:

cmdjob -jobname:"Backburner-job-nickname" -priority:50 -manager:2002:c01b:5753::c01b:5753 -port:3234 -servers:ZRCRA062799 -taskList:"D:\Current Projects\TerragenTests\TaskList_Maker1\TaskList Maker-120.txt" -taskName:2 "C:\Program Files\Planetside Software\Terragen 3\tgdcli.exe" "D:\1100635\Documents\Terragen Files\GettingStarted\GettingStarted-022.tgd" -r -f %%tp1