Animation crashes after frame 1 (with workaround!)

Started by bigben, March 03, 2007, 04:41:45 AM

Previous topic - Next topic

bigben

<update> For a better workaround, see this post using a camera-based mask. http://forums.planetside.co.uk/index.php?topic=972.msg10345#msg10345. Note that this approach also appears to be restricted by the population limit alluded to here</update>

I'm trying to render a panorama of my Tetons WIP with high quality settings (see Image Sharing) but I can't get past frame 1. :(

My initial problems related to RAM restrictions, but after reading the excellent post on RAM issues here and much testing I came up with an image size/camera fov that would enable me to actually render the images manually. (100 x 100 pixels, 10 degrees fov). Using this the RAM usage starts around 800Mb and goes up to around 1.1Gb depending on how many objects are in the frame.

Setting up the camera rotations and stitching script are pretty easy, so I don't care that I'm rendering so many frames, but when I try to render a sequence of images TG crashes without error message after saving frame 1.  I'm now up to frame 11 in this session, changing frames manually and rednering individual frames so i'm pretty confident it's not a rendering issue. I also started at the zenith so there wasn't much to render, but it doesn't make any difference where I start.

Aside on RAM usage: after connecting and opening IE, the RAM usage for TG dropped to 150Mb and started on up again...and it's still rendering nicely. A bit odd, but irrelevant here.

Aside on sky render times: It was interesting to note the render times for a small section of sky at various pitch angles. The render times increase with pitch angles, being quite long at the zenith.  May be something to keep in mind if you're trying to get a quick render. (Probably a side effect of remapping to the top of the sphere)

bigben

#1
Some additional info.

I rendered a small sequence OK on another computer using a default TGD so that's definitely working. The RAM usage of individual frames is well below my "crash threshold".

Attached is a clip file of my camera and render settings if anyone else wants to test it. I'm up to frame 29 doing manual renders without any crashes. Render times are between 6 minutes to 4 hours, the long times due mainly to objects (population object render quality at maximum).

If anyone has any ideas I'm all ears. 614 frames of manual rendering is going to be a pain.   I'll even throw in a script to stitch the render settings above ;)

Edit:  Stopped manually rendering frames to run some more tests.


  • Dropped render quality down, still crashes after saving 1 frame
  • Rendered a basic TGD using the attached clip file on the same machine. Works OK (still rendering anyway)
  • I'll try adding a population and water separately to see if I can isolate which element causes this.

Otherwise if any of the developers want to test my project I can make the files available.

Edit 2:

  • Disabled all populations and dropped render quality: OK
  • Increased render quality to previous settings: OK

Trying to add populations 1 by 1 (or source by source if the first one fails). At least it's not a water issue :)


  • Add grass: Crashed. Originally an OBJ with TG surfaces, translucency and reflectivity
  • Add blue spruce: Crashed. Originally an XFrog public OBJ, requires rotating around X-axis -90°. Image map only with translucency and reflectivity
  • Add grand fir (x3): Crashed. Xfrog TGO. No alterations (Image map only, reflectivity and translucency = 0).
  • Basic terrain + grand fir, without density shader: OK :)
  • Original TGD, minus density shaders on populations: Crashed :(.
  • Original TGD minus all surfaces + populations without density shaders: Crashed
  • Reduced area of populations and gradually introduced surfaces and populations back in to try and find a point at which it crashes. Reduced height and width of all populations to 500m, with density shaders active: OK! (original populations 20km square except grass which is 1km)
This is getting promising, as I know it's not my objects or TGD (which render manually frame by frame anyway)  From watching the progress of frame changes several times now, the crash occurs shortly after the "Populator inserting...." appears. My best guess at this stage is that the sheer number of objects is causing some sort of timing problem when rendering a sequence. Why this happens when rendering a sequence and not when manually changing frames is for someone else to figure out  ;).

Oshyan

We've seen some issues with sequence output in the past, but it's never been consistent enough to nail down what causes it. Since then some options have been put in that allow choosing whether populations are recalculated every frame. Given the possible role of populations as noted in your last post, it would be interesting to know what setting you have chosen for this, and whether switching it to the opposite has any effect.

Your investigations are much appreciated and given your familiarity with the scene, you are probably the best person to find out what exactly is happening if you are willing to continue experimenting. If you'd like us to take a look at the scene files you can email them to the standard support @ planetside address.

- Oshyan

bigben

Thanks Oshyan.

I've edited my last reply after yours, with my final findings.  I will try re-populating every frame. This will add about 12 minutes per frame which doesn't really worry me when 1/3 of the frames take > 3 hours. I'll let you know what happens.

Oshyan

Thanks Ben. If you can post any further findings in a reply that would be helpful.

- Oshyan

bigben

I set all populations to repopulate every frame but it still crashed after frame 1. Here's a summary and a repeatability test that will hopefully provide some clues.

Summary:

  • Rendering a sequence crashes at the start of frame 2 shortly after "Populator inserting...." appears
  • The project renders successfullly when manually changing frames between renders, without repopulating objects.
  • Changing the setting of "Repopulate every frame" has no effect
  • Reducing the size of the population fixes the problem
  • Aborting a re-population will crash a sequence render, while a static render normally continues with currently populated objects. (happened 2 out of 2)

Attached is a TGD I created to try and find a limit to this problem. It uses the public XFrog "CL02a_Grand_Fir.tgo" On my system, a Quick render sequence runs OK. Change the "area length a" to 3700 for the population and it crashes. Rendering individual frames works fine.

bigben

#6
I forgot one obvious test. Is the problem related to the total number of objects in a project or the maximum number of objects in any single population ???

The example above successfully rendered a population 3000 x 3500 (10.5km2) with a spacing of 10. I ran another test on another computer with only 1Gb RAM with 9 populations of 2000 x 2000 with the same spacing. That's 18km2, so it would appear that the problem is related to the number of objects in a population rather than the total number of objects in all of the populations.

This should provide an extra clue into tracking down the problem.

Suggested workaround

Giving a small safety margin I'll try calculating the maximum size of a particular population using:

(length x width)/spacing2 < 100,000

... and then just tiling populations to make up any shortfall in coverage.

Fingers crossed  :)

Update: The grass spacing in my project is 0.5. Using square tiles to simplify the above formula to calculate the maximum width of a population:   

Max width = sqrt (spacing2 x 100,000) = 158m in this case.

I set up 4 populations with a width of 150m and the sequence doesn't crash  ;D. I've set up an excel spreadsheet to give me a maximum width and to generate coordinates for tiled populations to replace my large populations. Now to replace all of my populations and try again.

cyphyr

I have found a simmilar problem with crashing after 1 frame, even on very light settings. I was trying to get a population to "grow" over time, using the "Colour Offset" of a Power Fractle in the Fractle Breakup channel keyframed over 100 frames from -0.7 to 0.7. Sure enough the population does grow over properly but crashes on render at frame 2. Its completely repeatable and I have made a variety of tries with the same result. Could be a sweet effect if it worked
Richard Fraser
www.richardfraservfx.com
https://www.facebook.com/RichardFraserVFX/
/|\

Ryzen 9 5950X OC@4Ghz, 64Gb (TG4 benchmark 4:13)

bigben

#8
Thanks for confirming the problem.

I tried this formula on my real project (using only my grass population) and I think it was an oversimplification. I should have used spacing2 which makes more sense since we are talking about area. This would also explain why my last test failed ;)   Adjusting the formula above and retrying.

I'm sure they'll be working on this one but the more we can find out about the problem the easier it should be to isolate (and then fix)

Quote from: cyphyr on March 05, 2007, 12:36:58 PM
I was trying to get a population to "grow" over time, using the "Colour Offset" of a Power Fractle in the Fractle Breakup channel keyframed over 100 frames from -0.7 to 0.7. Sure enough the population does grow over properly but crashes on render at frame 2. Its completely repeatable and I have made a variety of tries with the same result. Could be a sweet effect if it worked

As a suggestion for this, I'd probably use the "Colour adjust shader" for this.  You could then use a combination of black level and gamma to control the population density as it grows. This should provide finer control and flexibility of the density shader at various stages.

skyasay

Bigben, Thanks for putting so much time into the problem.  I am working on rendering a sequence with a simple camera rotation as a way to tile together a larger image.  I am using a near orthographic fov so that I can stitch the tiles together as one large none panoramic image(9800x4800pixels).  I have run into the same problem with the sequence crashing at the start of frame 2.
My scene includes:
an obj of the buildings in downtown Phoenix
DEM Hightfeild of the local terrain
trees
and house modules used to populate the sarounding city.

I will try to create a series of smaller populations to validate your findings.

skyasay

After looking at the first images I had finished rendering, I realized that there was a motion blur setting on the camera.  After turning this to 0, the sections that were crashing before are now on to frame 2.  I will change this post if they for some reason crashing at a later point during frame 2.  This is with no modification to my populations ( 3 populations all over 10,000 by 10,000 meters)
Sky Asay

bigben

Quote from: skyasay on March 06, 2007, 02:34:07 PM
After looking at the first images I had finished rendering, I realized that there was a motion blur setting on the camera.  After turning this to 0, the sections that were crashing before are now on to frame 2.  I will change this post if they for some reason crashing at a later point during frame 2.  This is with no modification to my populations ( 3 populations all over 10,000 by 10,000 meters)
Sky Asay

I already had to turn motion blur off as it's not a real animation ;). What is the object spacing of your populations? 

bigben

Being the nerd that I am at times, I created a database tool to generate population tiles and place them nicely in the node network. This worked really well, but unfortunately after I had tiled 3 of my populations I ran into RAM issues opening the file ( I was close enough to the edge beforehand anyway)

I'll keep all of this in mind for my next project and wait for the next update.  If the CLI is nearly fixed, this might provide a good short term workaround for this issue as I've had no trouble manually selecting and rendering frames.

Oshyan

The CLI should be fixed in the next update, but this animation rendering bug with populations likely won't be. Hopefully the CLI workaround will do until the following update.

- Oshyan

bigben

#14
That's good news. That will make this workaround  a beefed up repopulate every frame and I can't think of any reason why it shouldn't work.  It'll also mean I can stitch these panoramas on the fly which will save a lot of mucking around (and disk space).

<update> and indeed it does work... :) So far the only thing that's stopped my render was a Windows Update-induced reboot at 3am  ;)  There doesn't appear to be any extra load to exiting/reopening TGDCLI compared to repopulating every frame in the GUI. Let the fun begin   ;D