<update> For a better workaround, see this post using a camera-based mask. http://forums.planetside.co.uk/index.php?topic=972.msg10345#msg10345 (http://forums.planetside.co.uk/index.php?topic=972.msg10345#msg10345). Note that this approach also appears to be restricted by the population limit alluded to here</update>
I'm trying to render a panorama of my Tetons WIP with high quality settings (see Image Sharing) but I can't get past frame 1. :(
My initial problems related to RAM restrictions, but after reading the excellent post on RAM issues here and much testing I came up with an image size/camera fov that would enable me to actually render the images manually. (100 x 100 pixels, 10 degrees fov). Using this the RAM usage starts around 800Mb and goes up to around 1.1Gb depending on how many objects are in the frame.
Setting up the camera rotations and stitching script are pretty easy, so I don't care that I'm rendering so many frames, but when I try to render a sequence of images TG crashes without error message after saving frame 1. I'm now up to frame 11 in this session, changing frames manually and rednering individual frames so i'm pretty confident it's not a rendering issue. I also started at the zenith so there wasn't much to render, but it doesn't make any difference where I start.
Aside on RAM usage: after connecting and opening IE, the RAM usage for TG dropped to 150Mb and started on up again...and it's still rendering nicely. A bit odd, but irrelevant here.
Aside on sky render times: It was interesting to note the render times for a small section of sky at various pitch angles. The render times increase with pitch angles, being quite long at the zenith. May be something to keep in mind if you're trying to get a quick render. (Probably a side effect of remapping to the top of the sphere)
Some additional info.
I rendered a small sequence OK on another computer using a default TGD so that's definitely working. The RAM usage of individual frames is well below my "crash threshold".
Attached is a clip file of my camera and render settings if anyone else wants to test it. I'm up to frame 29 doing manual renders without any crashes. Render times are between 6 minutes to 4 hours, the long times due mainly to objects (population object render quality at maximum).
If anyone has any ideas I'm all ears. 614 frames of manual rendering is going to be a pain. I'll even throw in a script to stitch the render settings above ;)
Edit: Stopped manually rendering frames to run some more tests.
- Dropped render quality down, still crashes after saving 1 frame
- Rendered a basic TGD using the attached clip file on the same machine. Works OK (still rendering anyway)
- I'll try adding a population and water separately to see if I can isolate which element causes this.
Otherwise if any of the developers want to test my project I can make the files available.
Edit 2:
- Disabled all populations and dropped render quality: OK
- Increased render quality to previous settings: OK
Trying to add populations 1 by 1 (or source by source if the first one fails). At least it's not a water issue :)
- Add grass: Crashed. Originally an OBJ with TG surfaces, translucency and reflectivity
- Add blue spruce: Crashed. Originally an XFrog public OBJ, requires rotating around X-axis -90°. Image map only with translucency and reflectivity
- Add grand fir (x3): Crashed. Xfrog TGO. No alterations (Image map only, reflectivity and translucency = 0).
- Basic terrain + grand fir, without density shader: OK :)
- Original TGD, minus density shaders on populations: Crashed :(.
- Original TGD minus all surfaces + populations without density shaders: Crashed
- Reduced area of populations and gradually introduced surfaces and populations back in to try and find a point at which it crashes. Reduced height and width of all populations to 500m, with density shaders active: OK! (original populations 20km square except grass which is 1km)
This is getting promising, as I know it's not my objects or TGD (which render manually frame by frame anyway) From watching the progress of frame changes several times now, the crash occurs shortly after the "Populator inserting...." appears. My best guess at this stage is that the sheer number of objects is causing some sort of timing problem when rendering a sequence. Why this happens when rendering a sequence and not when manually changing frames is for someone else to figure out ;).
We've seen some issues with sequence output in the past, but it's never been consistent enough to nail down what causes it. Since then some options have been put in that allow choosing whether populations are recalculated every frame. Given the possible role of populations as noted in your last post, it would be interesting to know what setting you have chosen for this, and whether switching it to the opposite has any effect.
Your investigations are much appreciated and given your familiarity with the scene, you are probably the best person to find out what exactly is happening if you are willing to continue experimenting. If you'd like us to take a look at the scene files you can email them to the standard support @ planetside address.
- Oshyan
Thanks Oshyan.
I've edited my last reply after yours, with my final findings. I will try re-populating every frame. This will add about 12 minutes per frame which doesn't really worry me when 1/3 of the frames take > 3 hours. I'll let you know what happens.
Thanks Ben. If you can post any further findings in a reply that would be helpful.
- Oshyan
I set all populations to repopulate every frame but it still crashed after frame 1. Here's a summary and a repeatability test that will hopefully provide some clues.
Summary:
- Rendering a sequence crashes at the start of frame 2 shortly after "Populator inserting...." appears
- The project renders successfullly when manually changing frames between renders, without repopulating objects.
- Changing the setting of "Repopulate every frame" has no effect
- Reducing the size of the population fixes the problem
- Aborting a re-population will crash a sequence render, while a static render normally continues with currently populated objects. (happened 2 out of 2)
Attached is a TGD I created to try and find a limit to this problem. It uses the public XFrog "CL02a_Grand_Fir.tgo" On my system, a Quick render sequence runs OK. Change the "area length a" to 3700 for the population and it crashes. Rendering individual frames works fine.
I forgot one obvious test. Is the problem related to the total number of objects in a project or the maximum number of objects in any single population ???
The example above successfully rendered a population 3000 x 3500 (10.5km2) with a spacing of 10. I ran another test on another computer with only 1Gb RAM with 9 populations of 2000 x 2000 with the same spacing. That's 18km2, so it would appear that the problem is related to the number of objects in a population rather than the total number of objects in all of the populations.
This should provide an extra clue into tracking down the problem.
Suggested workaround
Giving a small safety margin I'll try calculating the maximum size of a particular population using:
(length x width)/spacing2 < 100,000
... and then just tiling populations to make up any shortfall in coverage.
Fingers crossed :)
Update: The grass spacing in my project is 0.5. Using square tiles to simplify the above formula to calculate the maximum width of a population:
Max width = sqrt (spacing2 x 100,000) = 158m in this case.
I set up 4 populations with a width of 150m and the sequence doesn't crash ;D. I've set up an excel spreadsheet to give me a maximum width and to generate coordinates for tiled populations to replace my large populations. Now to replace all of my populations and try again.
I have found a simmilar problem with crashing after 1 frame, even on very light settings. I was trying to get a population to "grow" over time, using the "Colour Offset" of a Power Fractle in the Fractle Breakup channel keyframed over 100 frames from -0.7 to 0.7. Sure enough the population does grow over properly but crashes on render at frame 2. Its completely repeatable and I have made a variety of tries with the same result. Could be a sweet effect if it worked
Richard Fraser
Thanks for confirming the problem.
I tried this formula on my real project (using only my grass population) and I think it was an oversimplification. I should have used spacing
2 which makes more sense since we are talking about area. This would also explain why my last test failed ;) Adjusting the formula above and retrying.
I'm sure they'll be working on this one but the more we can find out about the problem the easier it should be to isolate (and then fix)
Quote from: cyphyr on March 05, 2007, 12:36:58 PM
I was trying to get a population to "grow" over time, using the "Colour Offset" of a Power Fractle in the Fractle Breakup channel keyframed over 100 frames from -0.7 to 0.7. Sure enough the population does grow over properly but crashes on render at frame 2. Its completely repeatable and I have made a variety of tries with the same result. Could be a sweet effect if it worked
As a suggestion for this, I'd probably use the "Colour adjust shader" for this. You could then use a combination of black level and gamma to control the population density as it grows. This should provide finer control and flexibility of the density shader at various stages.
Bigben, Thanks for putting so much time into the problem. I am working on rendering a sequence with a simple camera rotation as a way to tile together a larger image. I am using a near orthographic fov so that I can stitch the tiles together as one large none panoramic image(9800x4800pixels). I have run into the same problem with the sequence crashing at the start of frame 2.
My scene includes:
an obj of the buildings in downtown Phoenix
DEM Hightfeild of the local terrain
trees
and house modules used to populate the sarounding city.
I will try to create a series of smaller populations to validate your findings.
After looking at the first images I had finished rendering, I realized that there was a motion blur setting on the camera. After turning this to 0, the sections that were crashing before are now on to frame 2. I will change this post if they for some reason crashing at a later point during frame 2. This is with no modification to my populations ( 3 populations all over 10,000 by 10,000 meters)
Sky Asay
Quote from: skyasay on March 06, 2007, 02:34:07 PM
After looking at the first images I had finished rendering, I realized that there was a motion blur setting on the camera. After turning this to 0, the sections that were crashing before are now on to frame 2. I will change this post if they for some reason crashing at a later point during frame 2. This is with no modification to my populations ( 3 populations all over 10,000 by 10,000 meters)
Sky Asay
I already had to turn motion blur off as it's not a real animation ;). What is the object spacing of your populations?
Being the nerd that I am at times, I created a database tool to generate population tiles and place them nicely in the node network. This worked really well, but unfortunately after I had tiled 3 of my populations I ran into RAM issues opening the file ( I was close enough to the edge beforehand anyway)
I'll keep all of this in mind for my next project and wait for the next update. If the CLI is nearly fixed, this might provide a good short term workaround for this issue as I've had no trouble manually selecting and rendering frames.
The CLI should be fixed in the next update, but this animation rendering bug with populations likely won't be. Hopefully the CLI workaround will do until the following update.
- Oshyan
That's good news. That will make this workaround a beefed up repopulate every frame and I can't think of any reason why it shouldn't work. It'll also mean I can stitch these panoramas on the fly which will save a lot of mucking around (and disk space).
<update> and indeed it does work... :) So far the only thing that's stopped my render was a Windows Update-induced reboot at 3am ;) There doesn't appear to be any extra load to exiting/reopening TGDCLI compared to repopulating every frame in the GUI. Let the fun begin ;D
Quote from: Oshyan on March 04, 2007, 06:57:52 PM
We've seen some issues with sequence output in the past, but it's never been consistent enough to nail down what causes it. Since then some options have been put in that allow choosing whether populations are recalculated every frame. Given the possible role of populations as noted in your last post, it would be interesting to know what setting you have chosen for this, and whether switching it to the opposite has any effect.
Your investigations are much appreciated and given your familiarity with the scene, you are probably the best person to find out what exactly is happening if you are willing to continue experimenting. If you'd like us to take a look at the scene files you can email them to the standard support @ planetside address.
- Oshyan
-------------------------------------------------------------------------------------------------------
NO PUN intended, but why should us users of TG2 should share info. as to how to do this and that with TG2??. There should be a COMPLETE tutorial on TG2. Do we agree on this?? Here we are.. I see people here sharing their findings when the makers and contributors of this program should come up with a COMPLETE tutorial.
This new program has jumped the gun as far as I am concerned... There are a LOT of inefficiencies still and I hope that the people who are responsible for this program are doing something about it to complete this program like it is supposed to be completed.
Rendering a composition that takes over 50 hours to render and only a few seconds to look at it in my book is completely ridiculous.
I like TG2, don't misunderstand me.... I love the potentials of it because of the clouds that can be produced and also the terrains that can be rendered, but I will be happy to see an "OFFICIAL, DETAILED, and FRIENDLY USER tutorial" instead of this confusing and yet "helping hints".
Invest more time on this beautiful program and it's potentials, pretty soon it will be over a year since it came out. I am the kind of person who tells it like it is, and so far I just see it like it is. Let me know just how I am wrong. I also like your opinion on this matter.
Sincerely,
Mario
Hi Mario,
We made the decision to release our current work on Terragen 2 nearly a year ago as a "Technology Preview" knowing that the community was eager to see what the product could be capable of and to start learning its new, powerful functionality and way of working. We have always tried to be clear that the product is in a pre-release state, not yet completed and still needing work, especially in the area of render time optimization and user interface. Over the months since that release we have worked on many areas and made a number of improvements and we are now working on some very significant improvements to the renderer which should improve speed and reduce render errors, as well as introduce several important new features. This is all part of the regular product development cycle that we have always envisioned.
As part of this development process the documentation has necessarily been incomplete. After all if the product is not yet finished, how can you fully document its workings? Many changes have been made in functionality since the original Technology Preview release that would have necessitated changes to any existing documentation. We do intend to provide both a complete Node Reference (explaining all the functionality and settings for every node) as well as a User Guide.
What we are not intending to do is provide "tutorials" on creating every type of feature or scene you might want to create. TG2 is enormously flexible and covering even a fraction of its capabilities as realized in an end product would take more time than we can possibly give. Our approach to enabling our users is to provide complete information on the functioning of the program itself and guidance on the actual process of scene creation in TG2, after which we hope the user will be equipped to experiment, discover and share their own methods for creating particular features and scene types. This is simply an issue of practicality in that we can only cover so much of the very broad topic of natural landscape generation and rendering in our own documentation.
We are currently working on completing the Node Reference and it will be made publicly available as soon as possible. We will then complete and expand the existing "tutorial-style" documentation. We hope that some of the more profficient users will continue to share their techniques with others, possibly documenting them formally in their own tutorials.
As TG2 grows in popularity we should see greater availability of resources, both free and commercial (e.g. terrain or texture packs for sale), and hopefully one day we'll even see books on working with TG2 just as there are for many other applications. That level of specific user-oriented writing is quite simply outside of realistic time and resource commitments for most companies, Planetside included.
I hope that the forthcoming documentation will provide you the information you need to really enjoy TG2.
- Oshyan
Thank you for your full explanation on TG2.
I will purchase the product after it has been completed, no sense on getting unnecessary headaches in the meantime. I believe that we should enjoy and have fun with Terragen ( I am having fun with the old version) and waiting 50 hrs. plus for a render or crashing while creating takes all the fun out of it.
Thanks again for your reply and I hope that you didn't think that I was too hard on you.
I only render very simple renders with TG2 demo. version and I have fun with it this way.
Mario
Your message was perfectly appropriate and not taken harshly. We appreciate honest feedback and constructive criticism. In general we're aware of the important issues with TG2 and are working to correct them, within the limits of time, development resources and finances that a small company like ours has. I hope you will enjoy the improvements in coming updates. :)
- Oshyan
Mario,
Can I just chime in and say that TG2 is a ground up rewrite of Terragen, and as such is the foundation upon which all subsequent versions will be built and as such needs to be done in a considered manor.
Terragen has always had a strong community, long before these forums opened a community that has always wanted to provide help and advice to its fellow members and shall continue to do so, the days of when the onus to provide every thing strictly by the software vendor alone are pretty much over and may one day go the way of dodo's, dinosaurs and...roto-scoping. With the advent of the Internet and the advantage's it brings for interaction and communication there has been a para dine shift in the way that software could and is been developed, that is to say that the end users get to have more say in what goes into the product an example of this model would be the modeling program Silo 3D made by NeverCenter and here as well.
There are software publishers that do not have this model and provide no means for community feedback on their products but that is at the end of the day the way that their business model is.
In conclusion then TG2 and those iterations that come after it are like the road less traveled, that is to say it is about (Or should be) the journey its self, not the destination, why struggle on alone when a friend or several can lead a hand?
In the end TG2 like any software will ever stand or fall by its users who will help keep things moving forward into the uncertain future, for the here and now we stand in the present to understand where we are we need to understand the past, what was right and what was not and only by doing so may way know what direction the future may bring, what ever that future is..its some thing to look forward to.
Regards to you.
Cyber-Angel