TGD rendering helper tool

Started by neon22, November 25, 2007, 05:19:48 AM

Previous topic - Next topic

neon22

Attached (and in file sharing) is a Pythoncard program to help automating rendering bundles of TGD files.
This is Version 3.

This allows creation of panoramas as well as tiled renders (with overlaps).
The tiles are well named and  Imagemagick scripts are created to composite the tiles into final images.
The panorama is also automatically made using Imagemagick scripts, if Nona (from Hugin project) is downloaded.
Also does animations.

It will only work for purchased versions of TG2 as it requires the command line option to work.

For Panoramas I am using Nona instead of PTStitcher. Big Ben supplied an excellent workflow and scripts using PTStitcher which I have learnt a lot from. http://forums.planetside.co.uk/index.php?topic=853.0 http://forums.planetside.co.uk/index.php?topic=972.0
Thanks BigBen. I hope this tool helps with the tiled image collating.

Screenshot below.
Please feel free to use and abuse this code.
Please fell free to make suggestions for improvements of any kind.

Installing
Alas - I have not packaged it up in an installer. sorry about that :-(
You need to load Pythoncard and its dependencies (Python and wxpython) as well as Imagemagick, Hugin

I built it using stable features of python 2.4 and wxpython 2.8
Later versions should work. earlier versions may not. (You must not use wxpython 2.6 - notebook is faulty)

Get Python from http://www.python.org/download/
   choose the latest or I used 2.4 (because I already had it)
Get wxpython from: http://www.wxpython.org/download.php#binaries
   Use a matching installer for your version of Python
Get Hugin from: http://hugin.sourceforge.net/download/
   we only use Nona (replaces PTStitcher for panoramas) but its all bundled together.
Get Imagemagick from http://www.imagemagick.org/script/binary-releases.php#windows
   the 16bit version does not support exr :-( so choose either the 16 or 8bit windows version
Get PythonCard 0.8.2 from http://pythoncard.sourceforge.net/
  - Specifically download from: http://sourceforge.net/project/showfiles.php?group_id=19015
  - Installation Instructions (seven steps including dependent packages) are here: 
    http://pythoncard.sourceforge.net/windows_installation.html

Install Python  ( I used 2.4 )
Install wxpython  ( I used 2.8.7.1 )
Install Pythoncard  ( I used 0.8.2 )
Install Imagemagick  ( I used 6.3.7 )
Install (just unpack really) Hugin ( I used 0.7 beta4)
  - see where Nona is located and enter its proper location into Batch_TG2.py
  - the location is defined in a global variable at the head of the file

Operation
Once PythonCard is installed.
  - Unzip the attached zipfile into any directory you like.
  - Double click on the file Batch_TG2.py

It will open a dialog prompting you to select a TGD file (or files).
   You may deactivate ones you do not wish to render.
Adding a Tile will allow you to tile or make a panorama
Scripts will be made that will composite the tiles into a single image.
  - Scripts will be named <imagename>_recombine.bat
Panoramas will create the recombine scripts, as well as a script that calls them all and creates an Equirectangular mapped image for a panorama.
   (Quicktime style but equally useful for environment maps)
  - Scripts will be named <imagename>_Collate_and_Build_Pano.bat
Panoramas can be viewed directly in panoglview (part of hugin install)

The log window will show you progress output from TGD as it runs in the background.
Animation is done by repeatedly opening the file and rendering single frames (see http://forums.planetside.co.uk/index.php?topic=954.0)

You can edit:
- size of image, quality settings, antialiasing, GI Settings, motion blur
- if animating - range, step-size
- many key settings - see screenshot

The program will make a temporary copy of your tgd file so as not to modify your original. It will prefix all these new TGD files with the prefix "tmp" so they are easy to find and delete after rendering.

Choose a rendering destination directory to collect frames - then render.
To recombine the tiles - run the script:  <imagename>_recombine.bat
To recombine tiles and build a panorama - run the script:  <imagename>_Collate_and_Build_Pano.bat
Your original rendered images will not be edited or deleted.
Your original tgd files will not be edited or deleted.
You will have to remove the surplus files manually (its just safer this way and they are well named)

Using "Batch" instead of "Render" will create the files to run but not run them.

Summary
I find it useful for:
- making up a batch of renders to run overnight (locally or on remote machines),
- setting off long animation sequences,
- making tiled renders for really large images (not HDR yet),
- Panoramas (not hdr yet)
Your original tgd files will not be edited or deleted.


User mods
You may wish to change the default starting directory.
   - Its defined at the head of the file: Batch_TG2.py
You may wish to change the "tmp" prefix for temporary TGD files.
   - Its defined at the head of the file: Batch_TG2_class.py
You will want to set the default location of the Nona.exe (if you make Panoramas)
   - Its defined at the head of the file: Batch_TG2.py

Next version may
   - do wedge tests of user selected parameters
   - support HDR options for panoramas and tiled images
   I need help (see about box) to get a version of Imagemagick compiled with OpenEXR support. Any takers ?
   If we can do that then I will alter it to make HDR panos and tiled images

Good luck - Please let me know how you get on.

Tangled-Universe

Oh man this is so useful! Now I can render an image + it's lens blur pass  Thank you very much for developing this tool!  :)
Will check it out as soon as my next project is finished.

Martin

Dark Fire

It looks very similar to my software, but probably with a better UI - I've been trying to get someone to add tabs to NSIS for ages now. >:(

Are there any plans to make this software work accross networks? If not, I'll just stick to networks. If so, I'll have to clean up my UI and start working on the frame-splitting feature...and I might try using a different language - PureBasic looks good...

gastar

hello,

I programm myself a network render client. At the moment it works over a local webserver/mysqlserver, so the master client load a tgd file, create many crop files and push the files to the webserver. The client look in the database for a project and take a crop file, render the crop area and send the picture back to the webserver. The idea is that the client works over the internet... currently I have only tested in my local lan. And I can say it works but the problem is, if you renderd a crop area, the GI effekt dosen't calculate correctly. Some parts are to dark, the result is a picture with dark area's... 

I wolud like create a big cloud image (4000x3000 pixel, detail 1, atmosphere sample128, 2x cloud sample 1024), on a single pc I must wait a month or more. So I wrote the client to use all my 4 PC's, PC's with dualcore cpu, start 2 clients... But the GI problem.... It was a nice idea!

I will finish the tool in the future, there some bugs at the moment.... When its down, I will post it here in the forum.

The tool works only with the deep version... it use the commandline parameter's

Gastar
current: AMD 4200+, 2 GB RAM, 500 GB, Geforce 7950 GX2
2010:    AMD Quadcore 3 Ghz, 4 GB RAM, more Space, ...
2015:    256 Core CPU, 16 GB RAM, more then 20 Terrabyte and a 64Bit Terragen Verison...

Dark Fire

It looks like I will have to get back to work at some point...

I thought about doing a similar thing, but I found two problems that were difficult to solve:
1. Not everyone has a server and database, which means that the software is either going to have to be limited to people with such facilities or someone is going to have to share. If the sharing option is chosen, that means either human time wasted processing applications to use the system or a completely automated system, which will obviously be open to abuse.
2. If a shared server is not used, the methods of communication between the client and server are going to have to be shared. That leaves everyone's personal servers open to abuse.

However, that's just me being fussy over how I use my extremely limited time. Such problems do not mean that cross-internet render farm software should not be written. I'm certainly interested in the project, as I would have undertaken it myself if I were more skilled.

Perhaps there are enough of these render manager projects now for a seperate forum to be started - perhaps a 'related software' forum.

neon22

Quote from: Dark Fire on November 30, 2007, 04:32:22 PM
Are there any plans to make this software work across networks? If not, I'll just stick to networks. If so, I'll have to clean up my UI and start working on the frame-splitting feature...and I might try using a different language - PureBasic looks good...

Hi Darkfire - Well its not supposed to supplant any other solutions out there. I've been watching your one for a while and hope its goes well. I am only doing it my way for the following reasons.
My initial problem was:

  • I've got a bunch of tgd files I want to render overnight on a single machine. (Exploring many aspects at the same time.)
  • I want to be able to wedge parameters. (and later view them simply too)
  • I need a solution for animation that calls one frame at a time (even though its a tremendous waste of time remaking the populations) because of a current bug where only a single frame is rendered.

So my solution creates a tgd file each frame and a batch file to execute it.
This happens to also be quite well suited to rendering across networks. However I haven't made anything for that.
In the back of my mind I consider something like "Condor" http://www.cs.wisc.edu/condor/ good for this aspect (Specifically their "Vanilla Universe" mechanism which does not require recompilation of TG2 with condor enhancements).
A standalone GUI managment tool for Condor rendering is GDIQueue but is designed for Max,Maya,XSI and seems a little limited but I haven't played with it http://www.cirquedigital.com/products/gdiqueue/index.html  so YMMV. The full package they sell costs 500 USD a seat but GDIQueue and Condor are free.
There appears to be an excellent (Java) job Monitor here: http://www.cs.wisc.edu/condor/tools/job-monitor/
Your solution is probably a good one to use also. I haven't got far enough yet to try it. :-(

My next version might do the tiling (and reconstruct the single image - or the panoramas BigBen has scripted).
But I think I should really get the Wedging working first... :-)

As for languages - well I'm an old Lisp hacker so Python is my preferred scripting language these days. Nice package implementation makes it easy to scale and I'm surprised at how simple and useful PyhtonCard is instead of using TCL (Tkinter) or wxwindows directly.

Anyway - that's my 2 cents....
Maybe we can pool ideas and come up with a useful system with a minimum of rewriting. I am sure some users have specific needs that are not addressed by our current plans either ...?

(attached is screenshot of next version of my Wedging UI - but code is not working yet)

lonewolf

This tool sounds like an excellent idea. I have a question for you though.
A while ago I was looking into the option of renderfarms etc, and came across a few ideas/instances of people writing software that could combine the computing power of many pc's to work with one renderer. If you guys could come up with something like this in youyr solution, we could be popping out large pictures in only a few hours instead of 15-20hrs+. If one had access to a small network one would have plenty of computing power.

Just an idea.

Marcos Silveira

Yeahhh, I have 40 PCs (each with 256Mb of RAM) desperately waiting to do that!!!!! ;D

rcallicotte

I could see poster-sized renders coming soon.


Quote from: lonewolf on December 27, 2007, 07:33:16 AM
This tool sounds like an excellent idea. I have a question for you though.
A while ago I was looking into the option of renderfarms etc, and came across a few ideas/instances of people writing software that could combine the computing power of many pc's to work with one renderer. If you guys could come up with something like this in youyr solution, we could be popping out large pictures in only a few hours instead of 15-20hrs+. If one had access to a small network one would have plenty of computing power.

Just an idea.
So this is Disney World.  Can we live here?

buchvecny

i guess i could use mothers comp for that :)

lonewolf

IBM Sun currently have a system for doing something like this. The software comes as client side and server side. The server knows how many clients are connected and there current work load. The client sends the job request to the server, and the server then assigns as many clients as it sees fit to complete the job that are currently idle. The job is then completed and the client is notified. Sounds simple, but getting it all to work accross the network and have support for terragen is the key...

neon22

If you follow the links to Condor you can see how they do it. It seems quite good and its one way to do it. There are many.
I think the mechanism you are hoping for is like a cluster where there is only one copy of the renderer program running but its multi-threaded across a very large number of virtual cores distributed on different machines. I am not sure TG2 is constructed this way.

But remember you'll need licenses to distribute rendering. The animation version of TG2 comes with several nodes worth of rendering licenses. We don't know what the license mechanism is yet. So its yet to be seen how to integrate it.

However if you have a license now - you can use my script to construct the batch files for each frame and manually distribute them to other machines.  BigBen also has a similar scheme and so does Darkfire.

I'm working on the tiling option for mine now.

P.S. The term Renderfarm was coined on the Symbolics graphics machines around 1986. In Chatsworth they manufactured them and burned them in ready for sale. The Graphics Division (SGD) people in LA would go out there and hook them up to render the large projects. (like Stanley and Stella http://www.imdb.com/title/tt0302371/). Chatsworth is out in the sticks some way and so they coined the term "RenderFarm" as it seemed appropriate.

Dark Fire

That's why there are so many of these render farms about - there are so many ways of tackling the problem of distributed rendering.

I've been thinking about the long-awaited frame-splitting feature in my software, and it's proving difficult to decide on the theory to use behind a fool-proof and error-proof system for splitting images. I think I've got the problem cracked now, but coding my solution is going to be a long process. Does anyone know of freely redistributable software that can stitch together images from the command line? I'd just rather avoid having to code that process too...

However, if I do have to tackle it all myself, at least I'll know I'm no longer wasting my time - experience in that area will probably help at university... :)

neon22

Quote from: Dark Fire on December 30, 2007, 08:59:34 PM

I've been thinking about the long-awaited frame-splitting feature in my software, and it's proving difficult to decide on the theory to use behind a fool-proof and error-proof system for splitting images. I think I've got the problem cracked now, but coding my solution is going to be a long process. Does anyone know of freely redistributable software that can stitch together images from the command line? I'd just rather avoid having to code that process too...


Working on it now. But imagemagick is excellent, has a python wrapper (and others), is command line capable, deals with OpenEXR format images as well as the regular ones.
http://www.imagemagick.org/script/index.php

I now have tiling working in my system - splits image up with overlaps (if desired), renders them and names images so you can re-assemble them manually if you like.
I will work on my Imagemagick script next (to auto-assemble) - or maybe I will finish what I started and enable the blending first.
Looks like holidays over soon. running out of time ;-)

EBAndrew

There isn't a way to switch cameras in your app, is there? I'm wondering if I could use it to render all six cameras in a file for a skybox.
-Andrew