Here is the latest version of this app that allows you to:
- render animation sequences (distributed to remote machines)
- create tiled large images
- render panoramas
- wedge node parameters (new)
Originally here: forum topic=2816.15 (http://forums.planetside.co.uk/index.php?topic=2816.15)
Instructions now located here:
http://neon.wireframe.biz/node/29 (http://neon.wireframe.biz/node/29)
Update 2 - corrected tgd_cli.exe location if using new beta
Update 3 - forgot new render params :(
Update 4 - fixed bug where all Tiled images were forced to be square
Looks interesting. Will give it a go after I reinstall windows (going 64bit).
Iain
Very nice. I'll try this. Thank you.
Looks very useful, I think I might have a project comming up that may be able to use this :)
Thanks
Oh dose this work in Vista64?
Richard
no idea :)
But I should think so... Let us know :)
Here is a sample panorama - not very big.
The master Render Camera's location defined the focal point for the renders.
The 6 frames were rendered - then stitched together - then projected into equirectangular format and saved as a Tiff. All done automatically by TG2_Batch.
I manually converted it to a jpg so it would be smaller for this page.
To see the image in a panoramic viewer you can:
- download FSPViewer - a large format standalone viewer
from here: http://www.fsoft.it/panorama/FSPViewer.htm (http://www.fsoft.it/panorama/FSPViewer.htm)
- view it in a webpage using the Java plugin: PTViewer
from here: http://www.fsoft.it/panorama/ptviewer.htm (http://www.fsoft.it/panorama/ptviewer.htm)
- download Panoglview - a hardware accelerated standalone viewer
from here: http://sourceforge.net/project/showfiles.php?group_id=77506&package_id=152817 (http://sourceforge.net/project/showfiles.php?group_id=77506&package_id=152817)
Is python meant to be this hard to install? Followed your insructions and get nothing.
I did download python26, which shouldn't be a problem, but wxpython only seems to be looking for python25 in the registry.
Iain
Decided to go back and download the older files, it seems to be working now.
But your app does not envoke tgd.exe or tgdcli.exe. is it meant to be in a certain directory, or is there a header that needs changing?
Iain
did you follow the instructions on the linked instructons page ?
For wxpython - you have to download the version that matches your python version.
You also have to install them in the right order... pythoncard will go into the current python version - if you have several on your machine.
I have tested installation on 3 machines - but I am sure I could write clearer instructions.
Can you determine where its all going wrong ?
During installation it was that wxpython would only look for pyton2.5 in the register, that was using all the latest dowload links from the python page. Put python2.5.2 on and it seems to be working now. They do say to use 2.6 but they haven't updated wx perhaps.
Anyway, your app loads, opens the tgd, changes settings, creates tile etc. But when I hit render, no successful renders out of n renders. Using the batch option creates the batch files for rendering in the same location as the tgd. Is there a line in one of your files that points to the TG2 installation folder? If I were to render with your batch files, how do I? In the command line text it says how to do it with a tgd file, is it similar?
Iain
yes there is a line where the executable is. Its at the top of Batch_TG2_class.py
tgd_executable_path = "C:/Program Files/Planetside Software/Terragen 2 Deep Edition (Beta)"
Also the start directory for looking for tgd files is at the top of the TG2_Batch.py file:
# starting directory for directory choosing dialog
homedir = "C:\\Documents and Settings\\User\\My Documents\\CTW\\Terragen etc\\programs"
(TG2_Batch.py is the main executable to start the program.)
Thanks for having a go at using it :-)
As for wxpython. This page suggests there are installers for 2.4, 2.5, 2.6 but YMMV
http://wxpython.org/download.php#binaries (http://wxpython.org/download.php#binaries)
Thanks for that neon. Am at work now so will try it later.
The problem seems to be that I am using vista 64, and thus there are 2 program files directories.
One for 32 bit and one for 64 bit programs.
Changing the path should start me making big renders.
Perhaps when you get closer to packaging it up you could include a batch file to ask for the installation directories of TG and nona, or make an installer to do the lot. Will start a render tonight and let you know how it goes.
Iain
Looks like a great tool. Unfortunately I'm using a Mac. Anychance of a Mac version?
Ok after I fixed the line pointing to the tg folder it seemed to work.
Did a quick test render at 1600x900 split into 6 tiles. Unfortunately it suffers from the GI inconsistency problem others have noticed.
Still, it did what you said it would and in reasonable time too. It was a little strange that I set it up to render at 1600x900 and it rendered at 1600x1600. Will be a bit of uneccesary rendering at higher resolutions. Here is the test image with only saving as a jpeg done after render and compile.
Iain
lol. As a follow up to my post near the end of the original thread, I am still working too hard to develop my own render farm manager (which, I must say, is looking increasingly like a more basic but friendly version of this TG2 Batch V4).
I can't be bothered to dig up my old threads again, so I'll just point out that the main thing holding my render farm manager back at the moment is the inability to stitch together portions of images (so there's no point, yet, in splitting images). Also, the inability to test a lot of the current features is both not helping development, and failing to keep me particularly interested.
DarkFire - Good to hear from you again.
Please feel free to take any of the code to use yourself.
All the imagemagick stuff is in there and does the recombining in batch scripts.
So its all just comandline Imagemagick "convert" commands. Very useful tool indeed :)
I just set up the crop sizes and then pass all the images to Imagmagick cleverness.
Please don't give up. Cheers...
Quote from: wackymidget on November 26, 2008, 06:02:09 AM
Looks like a great tool. Unfortunately I'm using a Mac. Anychance of a Mac version?
Well wxwindows runs on wxwidgets which is cross platrom.
python is cross platfrom. Imagemagick is available for Mac (I think)
The commandline Imagemagick scripts use Windows directory format like "c:\blah" but this should be only difference I think.
So I would suggest having a go and seeing how far you get.
Feel free to PM me with your experiments and I am happy to try to mod it to work for mac if possible.
Quote from: lonewolf on November 26, 2008, 06:47:47 AM
Ok after I fixed the line pointing to the tg folder it seemed to work.
Did a quick test render at 1600x900 split into 6 tiles. Unfortunately it suffers from the GI inconsistency problem others have noticed.
Yes. A TG2 problem. However BigBen was trying a blur radius of 800. I haven't experimented with this yet. YMMV
I don't have that in the GUI yet. You will need to set it in the original TGD
Quote
It was a little strange that I set it up to render at 1600x900 and it rendered at 1600x1600. Will be a bit of uneccesary rendering at higher resolutions. Here is the test image with only saving as a jpeg done after render and compile.
Hmmm. Not sure about that. Width and height are settable in top pane. Seems to work for me... Can you experiment a little more ?
If you turned panorama on then it forces the images to be square but that is all I can think of...
Quote from: neon22 on November 26, 2008, 03:38:04 PM
DarkFire - Good to hear from you again.
Please feel free to take any of the code to use yourself.
All the imagemagick stuff is in there and does the recombining in batch scripts.
So its all just comandline Imagemagick "convert" commands. Very useful tool indeed :)
I just set up the crop sizes and then pass all the images to Imagmagick cleverness.
Please don't give up. Cheers...
Cheers. I'll have a look at the imagemagick stuff and try to integrate it into my software (while, of course, significantly developing other features) over my Christmas 'holidays' (yes, I have a formal write-up of a physics experiment to do, some ML programming and undoubtedly other stuff that I haven't been told about yet).
I might start saving for an educational version of TG2 over Christmas too, so that I can finally test my software again - the only problem is that I don't naturally have any particular use for TG2 at the moment, so it's a bit of a random investment (especially for a student). Although, I am intrigued by the idea of making a render farm monitor for Windows Mobile (unfortunately, I can see that requiring more code than the rest of the render farm software, because I'd have to use a less friendly language)...
Quote from: neon22 on November 26, 2008, 03:45:52 PM
Quote from: lonewolf on November 26, 2008, 06:47:47 AM
It was a little strange that I set it up to render at 1600x900 and it rendered at 1600x1600. Will be a bit of unneccesary rendering at higher resolutions. Here is the test image with only saving as a jpeg done after render and compile.
Hmmm. Not sure about that. Width and height are settable in top pane. Seems to work for me... Can you experiment a little more ?
If you turned panorama on then it forces the images to be square but that is all I can think of...
Hah. Screwup on my part :(. This bug fixed in update 4 - see first post)
i've tried to use both tgd batch and erfm to do batch rendering . i don't need tiling or any special image manipulation, just the ability to batch render . with both of these tools, i've had issues getting them to work, so hopefully someone will be able to shed some light on it - i fully admit in advance that i could have been stupid and missed something ...
neon22:
tgd batch, despite apparently loading the correct project file, it renders the default terragen scene (blue sky with camera pointed downward at the flat grey terrain .
darkfire:
it's possible i'm just getting the syntax wrong, but erfm does not seem to want to pass the -o flag to terragen, as i should see it in the dosbox window it is missing .
both:
i believe i read someone had asked this question before but i am presuming there is not the ability to set/change the camera that is rendered because there is not a flag to pass to tgd_cli that specifies it ?
any help is greatly appreciated .
Quote from: nirokugraphic on January 12, 2009, 01:06:02 PM
neon22:
tgd batch, despite apparently loading the correct project file, it renders the default terragen scene (blue sky with camera pointed downward at the flat grey terrain .
My interface does not allow you to change the camera position. It assumes you have one or more tgd files already setup and is more of a batch organiser and renderer. The exception is the panorama which does override the camera orientation and makes 6 new views. (However it does use the camera position defined in the file.)
As for always getting the default tgd file and pointing down at the ground. This I cannot readily account for.
Can you please examine the TGD_batch.bat file
This is created by the program and is then called to start the render.
It should be referencing the tmp copy of the tgd you want rendered.
The path to terragen should also be correct.
Here is a sample:
set TERRAGEN_PATH=C:/Program Files/Planetside Software/Terragen 2 Deep Edition (Beta)
set RENDIR=C:\Documents and Settings\MarkS\Desktop\Terrains
cd "C:\Documents and Settings\MarkS\Desktop\Terrains"
"%TERRAGEN_PATH%\tgdcli" -p "tmpterrain_following_cloud.tgd" -hide -exit -r -f 1 -o "%RENDIR%\terrain_following_cloud.0001.bmp"The other thing the program does is make a copy of the original tgd and rename it tmp....tgd. Open this up in TG2 and see what it looks like.
One of these is at fault :(
Neon22: this script system is EXTREMELY AWESOME. Thank you very much for taking the time to make this, I think that Planetside should strive to achieve this level of functionality in Terragen 2 natively, as panoramas are a big reason that people purchase TG2. One thing to note is that it appears that if you use Alpha + Panorama, the nona scripts and batch files are not generated for the alphas, leaving you with 6 alpha images. Just a suggestion if you plan on doing a future release.
You are welcome Mecha. I see over 650 views of the script but I think the hassle of following my (complete and detailed) installation instructions puts people off using it.
But I find it incredibly useful - both the panoramas and the wedging of parameters.
Wish I had the time to make an installer for it - I should look into that again.
Quotebatch files are not generated for the alphas
(when I first started writing it the alphas were not so predictable - I should update it :))
you could easily manually do substitutions in a text editor for the alpha images and then re-run the batch files.
(One of the reasons I made it multi-stage - so it can be easily further modified and stripped apart)
Thanks for trying it :-)
Quote from: neon22 on November 24, 2008, 02:48:34 AM
To see the image in a panoramic viewer you can:
...
- download Panoglview - a hardware accelerated standalone viewer
from here: http://sourceforge.net/project/showfiles.php?group_id=77506&package_id=152817 (http://sourceforge.net/project/showfiles.php?group_id=77506&package_id=152817)
I would like to try a hardware accelerated pano viewer but I can't find a windows build of panoglview. The claim is that it is included in the Hugin installation but I can't find any sign of it on my pc.
thanks
I successfully ran a 4x4 tiled render, created a pano and viewed it with FPSviewer!
(GI was already turned off for this reason, so no seams either.)
Thanks so much for this script, this is definitely the kind of top level functionality that should be in Terragen. I just bought TG2 and have been despairing over the fact that I can't even run a decent-sized quality render without bucket errors. Hopefully this will fix that problem.
Is this the proper forum to discuss issues?
I'm running XP SP3.
-Selecting multiple files in the dialogue box only opens one of them. (not a deal killer)
-Loading a TGD file doesn't seem to give me a problem, considering my success above, but there is an error on the log screen I can't make sense of when I do. It might indicate something bad?
(first half of attached txt file)
-Add Wedge doesn't do anything. Just makes an empty tab. There are errors on the cmd output. If I create a 'Tile' tab first, the new 'Blend' tabs are filled with the same interface as the 'Tile' tabs. (same log output errors)
(remaining text of attached txt file)
-Can't close a TGD file, Tile, or Blend tab. They just accumulate.
-Can't save any of my settings, have to recreate them every time.
-Is there a way to save my settings to some kind of batch project?
-When wedging works for me, it's great! another must have! One thing I would really like to see though in the wedge panel annotations is the render time, since often I'm trading quality vs render time.
-Why are the recombine scripts and pano scripts not automatically called from the main script? I can't think of a typical reason for generating the tiles and NOT wanting to automatically merge them. For really slow operations I'd like to be able to come back and have final images waiting for me. (Although I don't know Python, I do know programming and could probably figure out how to do something so simple.)
Otherwise, this has the beginnings of a 'must have' script, and I mean 'really must have', like, "how does anybody use TG2 without it?" kinda usefulness. Thanks so much neon22 for making this available :D
Python command line output:
QuotePython 2.6.2 (r262:71605, Apr 14 2009, 22:40:02) [MSC v.1500 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
****************************************************************
Personal firewall software may warn about the connection IDLE
makes to its subprocess using this computer's internal loopback
interface. This connection is not visible on any external
interface and no data is sent to or received from the Internet.
****************************************************************
IDLE 2.6.2 ==== No Subprocess ====
>>>
Traceback (most recent call last):
File "C:\Python26\lib\site-packages\PythonCard\widget.py", line 408, in _dispatch
handler(background, aWxEvent)
File "D:\Downloads\Terragen\Accessories\TG2\TG2_BatchV4-4\Batch_TG2.py", line 98, in on_add_TGD_files_command
page.blend_categories = TG2_Param(self.current_tgd.tgd_dom.childNodes[0])
File "D:\Downloads\Terragen\Accessories\TG2\TG2_BatchV4-4\Batch_TG2_class.py", line 224, in __init__
group = [n for n in e if n[0] == "gui_group"][0][1]
IndexError: list index out of range
Traceback (most recent call last):
File "C:\Python26\lib\site-packages\PythonCard\widget.py", line 408, in _dispatch
handler(background, aWxEvent)
File "D:\Downloads\Terragen\Accessories\TG2\TG2_BatchV4-4\Batch_TG2_page.py", line 231, in on_wedgebook_pageChanged
self.components.wedgebook.getPage(w_page).loadwedgepageData(self.pageno, w_page)
File "D:\Downloads\Terragen\Accessories\TG2\TG2_BatchV4-4\Batch_TG2_Blends.py", line 50, in loadwedgepageData
self.blend_cat = self.GetParent().GetParent().GetParent().blend_categories
AttributeError: 'TGD_page' object has no attribute 'blend_categories'
>>>
Note: subsequently found I could make the wedge panel work if I deleted most of the items from my TGD file, so something I've used is making it choke.
-I'm keeping running annotations of my findings, bear with me...
-Found that my Merge Shader I'm using on a lake object doesn't Wedge if I use a Painted Shader in it's Blend Shader input. In fact I have to delete the painted shader completely, not just disconnect it.
Also, when I delete the Painted Shader and the wedge panel works, I find a terrain displacement shader elsewhere showing up under the water list in the wedge panel drop down list, which is totally wrong. I suspect a bug either with TGD or this python script.
-Removed EVERYTHING except the basics (atmo, background, planet, sunlight, cam, renderer) expect the painted shader, still no wedge. I tweaked every parameter possible in the shader, still no wedge....
...time passes....
-well, I've dug about as far in as I can go without knowing Python.
The open TGD error is definitely related to the failure of the wedge making operation.
The painted shader causes a problem. It also happens to be the last item defined in the blendables dictionary. I suspect an indexing or size problem.
...ok, I couldn't resist digging in, but this is getting ridiculous.
I got it to work, but only by deleting the painted_shader custom subsection in the .tgd file using a text editor! Come to think of it, that might have been my image.. :-[
Neon22: check it out.
I learned something interesting though. Painted masks are vector images, even though it's interacts like a vertex shader. My entire mask is stored in a short character string. Cool!
...after some more digging, I found an admittedly vulgar patch until the code is more elegantly updated.
I tested it on a wedge, worked great.
edit the following block of code around line 230 of Batch_TG2_class.py to include the line starting with if name == ''"
This assumes the painted shader image data is the only node with a null namestring.
for (node, e) in self.elements:
name = node.getAttribute("name")
if name == "": continue #skip the painted mask data
group = [n for n in e if n[0] == "gui_group"][0][1]
if group == '': group = lastgroup
self.groups[group].append([name, node.nodeName, e]) # to help debug
lastgroup = group
OK so the problem looks like I need to do substantial updating to deal with the new ops and settings as of TG2.0.
For me - and I am using python 2.5 and not 2.6 - I don't see the red or bue problems above and I can load multiple tgd files.
So something may be up with version skew of the components also. So I will have to update my entire environment I think.
But - as stated - I suspect the majority of the probs are due to the TG 2.0 changes. I hope to have some time next weekend to spend an hour or so on it and update it all.
Yes - noticed the painted shader's internals - its pretty cool actually. I am sure we will be able to do something interesting with that data :-)
Oh yeah - as to why the scripts are in two chunks - same reason as before - its more robust. I.e. I am not counting on all of the tiles being rendered in one continuous session or even on one machine. So the scripts to recombine are separate to the ones to initially render them.
I should add more doc I think...
Thanks for trying to use it - if even one person is having a go - it gives me some impetus to keep it up to date.
Cheers...
Now that I have Neon22's batch processor working and sitting around waiting for 40 hour stills, I find myself hungry to apply the three other PCs in my house.
Neon22: Have you thought about an approach that would work best to implement a way to farm out your batcher over a network?
Also, where are you getting your DOM model for TG2, or does that question even make sense? (XML is very confusing to me)
I was looking at my win_commandline.txt file and it seems out of date. Is there a new version available? Also, you are wedging all those parameters by cloning and modifying the actual TGD (cloned) file, right?
Quote from: shadowphile on September 07, 2009, 09:55:20 PM
Have you thought about an approach that would work best to implement a way to farm out your batcher over a network?
I make the files up for each frame - they have a complete tgd file for each frame. If you execute each one from commandline it will render a single frame (or tile if that's what you've done).
So all you have to do is copy the files (and any dependent files like referenced objs etc) to a new machine and execute them. (Please observe the TG2 license agreement for rendernodes).
So its a manual batch renderer. copy the batch and tgd files to a machine - execute them.
You could look at Darkfire's renderfarm for some possibilities - or use Condor. (both links on that page or thereabouts)
So I did not write a renderfarm manager - use Condor. YMMV
Quote from: shadowphile
Also, where are you getting your DOM model for TG2, or does that question even make sense? (XML is very confusing to me)
I was looking at my win_commandline.txt file and it seems out of date. Is there a new version available? Also, you are wedging all those parameters by cloning and modifying the actual TGD (cloned) file, right?
Yep - the xml file is parsed in a dom (yay python) then its easily accessable.
So that makes it easy for me to wedge a param and make a new tgd by writing out the dom (Yay python) :)
The code that does that is in the core file. The rest is UI and image compilation...
hey, that sounds easy. I could do file copies, monitoring and exe fire-up with just about any language, maybe even DOS, heh.
I'm unsure about how to determine dependent files to copy, but I'll figure it out.
Thanks, this is exciting, I can do this without big programming headaches.
Neon22: I'm still in need of HDR images so I've been tackling the ImageMagick HDR issue, but I'm not a (modern) programmer and I'm getting in over my head. I managed to compile ImageMagick with VSC++6, but the code is old and the OpenEXR files are much newer and at this point I'm getting overwhelmed. I'm attempted to rope in a friend who is much more savvy.
That issue will affect my ability to tile large HDR images.
I still have the other issue of getting Nona to tile HDR images for me.
I read through the latest Nona header file and it WILL recognize EXR files as an output.
I tweaked your nona script to pull in and output EXR files. It stitches great but the EXR is a faux HDR file. The dynamic range seems to be that of an LDR file. Converted to HDR, opened in HDRShop, dropped the exposure, my sun just dims along with everthing else. I've confirmed that the original EXR cubefaces don't do this.
Hmmm. I personally have noticed that the sun in exr renders from TG2 is not as bright as I would expect. Also there are black squares when viewed in an exr viewer. So not sure whats going on there.
But I think the dynamic range of EXR renders could be a lot higher in the "sun zone" than they currently are...
Personally this woudl help with using the results as envornment maps for lighting.
The sun in Terragen EXRs is usually too *bright* to be viewed in some EXR viewers, and this is why its pixels appear black. You can spread the energy around a little by increasing the diameter of the Terragen sun. This reduces the brightness of each pixel in the sun to something that the viewer can handle it, and also makes it easier to deal with as an environment map or IBL source. The total energy from the sun's pixels remains approximately the same regardless of the diameter you use.
Having said that, it's possible that one my constants is wrong for calculating the pixel brightness. It's not far out though.
Matt
I did notice the sun is maybe less bright than I expected, although it took me 17 stops down to get it to start dimming. It has an angular disk of 4, I think.
It was only after it was stitched by Nona that the sun behaved like an LDR, dimming even with one stop down and no discernible difference between the corona and the sun itself.
Hi Neon22. Have you examined the impact of 2.1 on your batch render? Your batch program seems to be the only way I can handle some of my scenes, despite my PC being maxed out. thanks.
Just got 2.1
will be checking over Xmas. but its sunny down here :-(
enjoy the sun!
when you get tired of it :), please give one of my previous suggestions a serious thought: render time info with each wedge image. I just finished a 59 hour render while on vacation and doing wedge studies now to specifically speed things up. (and using file timestamps, a real pain)
thanks
Hi Neon22. I attempted to use your batch file for my first ever! TG animation.
Question: Do I need the animation module for TG2 as well?
Do the channel files work in a non-animation-version of TG2?
For now I went with a wedge that moves the camera position, but that is capped at 100 frames and only allows the single translation. I couldn't see how the animation range controls would do anything in my file otherwise.
thanks
Hi, been a long time. Is it likely you will try to update your batcher for the latest TG releases?
thanks
Yes - it is - I am rewriting using Qt so it is easier to install and use.
UI is finished - done hooking it back up now.
Then I have to update the xml mappings for the new fields.
Also remove the old dead ones.
Working on a big Earth animation right now - so its not going so fast....