TG2 Batch V4: Panorama - Wedges - Large tiled Images - distrib Rendering

Started by neon22, November 22, 2008, 09:22:20 PM

Previous topic - Next topic

neon22

DarkFire - Good to hear from you again.
Please feel free to take any of the code to use yourself.
All the imagemagick stuff is in there and does the recombining in batch scripts.
So its all just comandline Imagemagick "convert" commands. Very useful tool indeed :)
I just set up the crop sizes and then pass all the images to Imagmagick cleverness.

Please don't give up. Cheers...

neon22

Quote from: wackymidget on November 26, 2008, 06:02:09 AM
Looks like a great tool. Unfortunately I'm using a Mac. Anychance of a Mac version?
Well wxwindows runs on wxwidgets which is cross platrom.
python is cross platfrom. Imagemagick is available for Mac (I think)
The commandline Imagemagick scripts use Windows directory format like "c:\blah" but this should be only difference I think.

So I would suggest having a go and seeing how far you get.
Feel free to PM me with your experiments and I am happy to try to mod it to work for mac if possible.

neon22

Quote from: lonewolf on November 26, 2008, 06:47:47 AM
Ok after I fixed the line pointing to the tg folder it seemed to work.
Did a quick test render at 1600x900 split into 6 tiles. Unfortunately it suffers from the GI inconsistency problem others have noticed.
Yes. A TG2 problem. However BigBen was trying a blur radius of 800. I haven't experimented with this yet. YMMV
I don't have that in the GUI yet. You will need to set it in the original TGD

Quote
It was a little strange that I set it up to render at 1600x900 and it rendered at 1600x1600. Will be a bit of uneccesary rendering at higher resolutions. Here is the test image with only saving as a jpeg done after render and compile.
Hmmm. Not sure about that. Width and height are settable in top pane. Seems to work for me... Can you experiment a little more ?
If you turned panorama on then it forces the images to be square but that is all I can think of...

Dark Fire

Quote from: neon22 on November 26, 2008, 03:38:04 PM
DarkFire - Good to hear from you again.
Please feel free to take any of the code to use yourself.
All the imagemagick stuff is in there and does the recombining in batch scripts.
So its all just comandline Imagemagick "convert" commands. Very useful tool indeed :)
I just set up the crop sizes and then pass all the images to Imagmagick cleverness.

Please don't give up. Cheers...

Cheers. I'll have a look at the imagemagick stuff and try to integrate it into my software (while, of course, significantly developing other features) over my Christmas 'holidays' (yes, I have a formal write-up of a physics experiment to do, some ML programming and undoubtedly other stuff that I haven't been told about yet).

I might start saving for an educational version of TG2 over Christmas too, so that I can finally test my software again - the only problem is that I don't naturally have any particular use for TG2 at the moment, so it's a bit of a random investment (especially for a student). Although, I am intrigued by the idea of making a render farm monitor for Windows Mobile (unfortunately, I can see that requiring more code than the rest of the render farm software, because I'd have to use a less friendly language)...

neon22

Quote from: neon22 on November 26, 2008, 03:45:52 PM
Quote from: lonewolf on November 26, 2008, 06:47:47 AM
It was a little strange that I set it up to render at 1600x900 and it rendered at 1600x1600. Will be a bit of unneccesary rendering at higher resolutions. Here is the test image with only saving as a jpeg done after render and compile.
Hmmm. Not sure about that. Width and height are settable in top pane. Seems to work for me... Can you experiment a little more ?
If you turned panorama on then it forces the images to be square but that is all I can think of...
Hah. Screwup on my part :(. This bug fixed in update 4 - see first post)

nirokugraphic

i've tried to use both tgd batch and erfm to do batch rendering . i don't need tiling or any special image manipulation, just the ability to batch render . with both of these tools, i've had issues getting them to work, so hopefully someone will be able to shed some light on it - i fully admit in advance that i could have been stupid and missed something ...

neon22:

tgd batch, despite apparently loading the correct project file, it renders the default terragen scene (blue sky with camera pointed downward at the flat grey terrain .

darkfire:

it's possible i'm just getting the syntax wrong, but erfm does not seem to want to pass the -o flag to terragen, as i should see it in the dosbox window it is missing .

both:

i believe i read someone had asked this question before but i am presuming there is not the ability to set/change the camera that is rendered because there is not a flag to pass to tgd_cli that specifies it ?


any help is greatly appreciated .


neon22

Quote from: nirokugraphic on January 12, 2009, 01:06:02 PM
neon22:
tgd batch, despite apparently loading the correct project file, it renders the default terragen scene (blue sky with camera pointed downward at the flat grey terrain .
My interface does not allow you to change the camera position. It assumes you have one or more tgd files already setup and is more of a batch organiser and renderer. The exception is the panorama which does override the camera orientation and makes 6 new views. (However it does use the camera position defined in the file.)

As for always getting the default tgd file and pointing down at the ground. This I cannot readily account for.
Can you please examine the TGD_batch.bat file
This is created by the program and is then called to start the render.
It should be referencing the tmp copy of the tgd you want rendered.
The path to terragen should also be correct.
Here is a sample:
set TERRAGEN_PATH=C:/Program Files/Planetside Software/Terragen 2 Deep Edition (Beta)
set RENDIR=C:\Documents and Settings\MarkS\Desktop\Terrains
cd "C:\Documents and Settings\MarkS\Desktop\Terrains"
"%TERRAGEN_PATH%\tgdcli" -p "tmpterrain_following_cloud.tgd" -hide -exit -r -f 1 -o "%RENDIR%\terrain_following_cloud.0001.bmp"

The other thing the program does is make a copy of the original tgd and rename it tmp....tgd. Open this up in TG2 and see what it looks like.
One of these is at fault :(

mecha

Neon22: this script system is EXTREMELY AWESOME. Thank you very much for taking the time to make this, I think that Planetside should strive to achieve this level of functionality in Terragen 2 natively, as panoramas are a big reason that people purchase TG2. One thing to note is that it appears that if you use Alpha + Panorama, the nona scripts and batch files are not generated for the alphas, leaving you with 6 alpha images. Just a suggestion if you plan on doing a future release.

neon22

You are welcome Mecha. I see over 650 views of the script but I think the hassle of following my (complete and detailed) installation instructions puts people off using it.
But I find it incredibly useful - both the panoramas and the wedging of parameters.
Wish I had the time to make an installer for it - I should look into that again.

Quotebatch files are not generated for the alphas
(when I first started writing it the alphas were not so predictable - I should update it :))
you could easily manually do substitutions in a text editor for the alpha images and then re-run the batch files.
(One of the reasons I made it multi-stage - so it can be easily further modified and stripped apart)

Thanks for trying it :-)

shadowphile

Quote from: neon22 on November 24, 2008, 02:48:34 AM
To see the image in a panoramic viewer you can:
...
  - download Panoglview - a hardware accelerated standalone viewer
      from here: http://sourceforge.net/project/showfiles.php?group_id=77506&package_id=152817


I would like to try a hardware accelerated pano viewer but I can't find a windows build of panoglview. The claim is that it is included in the Hugin installation but I can't find any sign of it on my pc.
thanks

shadowphile

#25
I successfully ran a 4x4 tiled render, created a pano and viewed it with FPSviewer!
(GI was already turned off for this reason, so no seams either.)
Thanks so much for this script, this is definitely the kind of top level functionality that should be in Terragen.  I just bought TG2 and have been despairing over the fact that I can't even run a decent-sized quality render without bucket errors.  Hopefully this will fix that problem.
Is this the proper forum to discuss issues?

I'm running XP SP3.

-Selecting multiple files in the dialogue box only opens one of them. (not a deal killer)

-Loading a TGD file doesn't seem to give me a problem, considering my success above, but there is an error on the log screen I can't make sense of when I do.  It might indicate something bad?
(first half of attached txt file)

-Add Wedge doesn't do anything.  Just makes an empty tab.  There are errors on the cmd output. If I create a 'Tile' tab first, the new 'Blend' tabs are filled with the same interface as the 'Tile' tabs. (same log output errors)
(remaining text of attached txt file)

-Can't close a TGD file, Tile, or Blend tab.  They just accumulate.
-Can't save any of my settings, have to recreate them every time.
-Is there a way to save my settings to some kind of batch project?
-When wedging works for me, it's great!  another must have!  One thing I would really like to see though in the wedge panel annotations is the render time, since often I'm trading quality vs render time.

-Why are the recombine scripts and pano scripts not automatically called from the main script?  I can't think of a typical reason for generating the tiles and NOT wanting to automatically merge them.  For really slow operations I'd like to be able to come back and have final images waiting for me.  (Although I don't know Python, I do know programming and could probably figure out how to do something so simple.)

Otherwise, this has the beginnings of a 'must have' script, and I mean 'really must have', like, "how does anybody use TG2 without it?" kinda usefulness.  Thanks so much neon22 for making this available :D

Python command line output:
QuotePython 2.6.2 (r262:71605, Apr 14 2009, 22:40:02) [MSC v.1500 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.

   ****************************************************************
   Personal firewall software may warn about the connection IDLE
   makes to its subprocess using this computer's internal loopback
   interface.  This connection is not visible on any external
   interface and no data is sent to or received from the Internet.
   ****************************************************************
   
IDLE 2.6.2      ==== No Subprocess ====
>>>
Traceback (most recent call last):
 File "C:\Python26\lib\site-packages\PythonCard\widget.py", line 408, in _dispatch
   handler(background, aWxEvent)
 File "D:\Downloads\Terragen\Accessories\TG2\TG2_BatchV4-4\Batch_TG2.py", line 98, in on_add_TGD_files_command
   page.blend_categories = TG2_Param(self.current_tgd.tgd_dom.childNodes[0])
 File "D:\Downloads\Terragen\Accessories\TG2\TG2_BatchV4-4\Batch_TG2_class.py", line 224, in __init__
   group = [n for n in e if n[0] == "gui_group"][0][1]
IndexError: list index out of range

Traceback (most recent call last):
 File "C:\Python26\lib\site-packages\PythonCard\widget.py", line 408, in _dispatch
   handler(background, aWxEvent)
 File "D:\Downloads\Terragen\Accessories\TG2\TG2_BatchV4-4\Batch_TG2_page.py", line 231, in on_wedgebook_pageChanged
   self.components.wedgebook.getPage(w_page).loadwedgepageData(self.pageno, w_page)
 File "D:\Downloads\Terragen\Accessories\TG2\TG2_BatchV4-4\Batch_TG2_Blends.py", line 50, in loadwedgepageData
   self.blend_cat = self.GetParent().GetParent().GetParent().blend_categories
AttributeError: 'TGD_page' object has no attribute 'blend_categories'

>>>

Note: subsequently found I could make the wedge panel work if I deleted most of the items from my TGD file, so something I've used is making it choke.
-I'm keeping running annotations of my findings, bear with me...
-Found that my Merge Shader I'm using on a lake object doesn't Wedge if I use a Painted Shader in it's Blend Shader input.  In fact I have to delete the painted shader completely, not just disconnect it.
Also, when I delete the Painted Shader and the wedge panel works, I find a terrain displacement shader elsewhere showing up under the water list in the wedge panel drop down list, which is totally wrong.  I suspect a bug either with TGD or this python script.
-Removed EVERYTHING except the basics (atmo, background, planet, sunlight, cam, renderer) expect the painted shader, still no wedge.  I tweaked every parameter possible in the shader, still no wedge....

...time passes....

-well, I've dug about as far in as I can go without knowing Python.
The open TGD error is definitely related to the failure of the wedge making operation.
The painted shader causes a problem.  It also happens to be the last item defined in the blendables dictionary.  I suspect an indexing or size problem.  

...ok, I couldn't resist digging in, but this is getting ridiculous.
I got it to work, but only by deleting the painted_shader custom subsection in the .tgd file using a text editor!  Come to think of it, that might have been my image.. :-[

Neon22: check it out.  

I learned something interesting though.  Painted masks are vector images, even though it's interacts like a vertex shader.  My entire mask is stored in a short character string.  Cool!

...after some more digging, I found an admittedly vulgar patch until the code is more elegantly updated.
I tested it on a wedge, worked great.

edit the following block of code around line 230 of Batch_TG2_class.py to include the line starting with if name == ''"
This assumes the painted shader image data is the only node with a null namestring.

        for (node, e) in self.elements:
            name = node.getAttribute("name")
            if name == "": continue  #skip the painted mask data
            group = [n for n in e if n[0] == "gui_group"][0][1]
            if group == '': group = lastgroup
            self.groups[group].append([name, node.nodeName, e]) # to help debug
            lastgroup = group

neon22

OK so the problem looks like I need to do substantial updating to deal with the new ops and settings as of TG2.0.
For me - and I am using python 2.5 and not 2.6 - I don't see the red or bue problems above and I can load multiple tgd files.
So something may be up with version skew of the components also. So I will have to update my entire environment I think.

But - as stated - I suspect the majority of the probs are due to the TG 2.0 changes. I hope to have some time next weekend to spend an hour or so on it and update it all.

Yes - noticed the painted shader's internals - its pretty cool actually. I am sure we will be able to do something interesting with that data :-)

Oh yeah - as to why the scripts are in two chunks - same reason as before - its more robust. I.e. I am not counting on all of the tiles being rendered in one continuous session or even on one machine. So the scripts to recombine are separate to the ones to initially render them.
I should add more doc I think...

Thanks for trying to use it - if even one person is having a go - it gives me some impetus to keep it up to date.
Cheers...

shadowphile

Now that I have Neon22's batch processor working and sitting around waiting for 40 hour stills, I find myself hungry to apply the three other PCs in my house.

Neon22: Have you thought about an approach that would work best to implement a way to farm out your batcher over a network?
Also, where are you getting your DOM model for TG2, or does that question even make sense?  (XML is very confusing to me)
I was looking at my win_commandline.txt file and it seems out of date.  Is there a new version available?  Also, you are wedging all those parameters by cloning and modifying the actual TGD (cloned) file, right?

neon22

Quote from: shadowphile on September 07, 2009, 09:55:20 PM
Have you thought about an approach that would work best to implement a way to farm out your batcher over a network?
I make the files up for each frame - they have a complete tgd file for each frame. If you execute each one from commandline it will render a single frame (or tile if that's what you've done).
So all you have to do is copy the files (and any dependent files like referenced objs etc) to a new machine and execute them. (Please observe the TG2 license agreement for rendernodes).

So its a manual batch renderer. copy the batch and tgd files to a machine - execute them.
You could look at Darkfire's renderfarm for some possibilities - or use Condor. (both links on that page or thereabouts)
So I did not write a renderfarm manager - use Condor. YMMV

Quote from: shadowphile
Also, where are you getting your DOM model for TG2, or does that question even make sense?  (XML is very confusing to me)
I was looking at my win_commandline.txt file and it seems out of date.  Is there a new version available?  Also, you are wedging all those parameters by cloning and modifying the actual TGD (cloned) file, right?
Yep - the xml file is parsed in a dom (yay python) then its easily accessable.
So that makes it easy for me to wedge a param and make a new tgd by writing out the dom (Yay python) :)
The code that does that is in the core file. The rest is UI and image compilation...


shadowphile

hey, that sounds easy.  I could do file copies, monitoring and exe fire-up with just about any language, maybe even DOS, heh.
I'm unsure about how to determine dependent files to copy, but I'll figure it out.
Thanks, this is exciting, I can do this without big programming headaches.