Please add resume function to render engine

[commorancy]

I am rendering very large images and they can take days to render.  Unfortunately, I don't have a second computer and need to use this one with TG2 in the background rendering.  This computer has 4 cores (core i7) with 8 threads available so there's plenty of extra unused processing power.  I have had two blue screens (for other unrelated reasons) on this system twice during the render and I have to completely start the render over each time.  I've already wasted about 2 days restarting this render.  The first blue screen happened almost 3/4 through the render.

So, what I suggest is that rendering engine should periodically write what's being rendered to disk with a restart marker containing information on how to resume the rendering already in progress (no recalc necessary.. just jump in and start right up).  This functionality has a lot of usefulness.  It allows you to recover from blue screen crashes.  It allows you to recover from TG2 crashes or errors (render thread problems).  It allows you to stop the render to install Windows updates and reboot.  It allows you to stop the render to install software and reboot.  You could even stop the render, move the files and resume the render on another machine with complete enough resume data.

Please add this functionality.

Thanks.

--
Brian

[Oshyan]

This is actually a very commonly requested feature. Unfortunately it's not nearly as simple to implement as it might seem. As a matter of fact the majority of render engines don't have this feature, even high-end professional systems like Vray or Mental Ray. There are of course some renderers that do allow this, but it would require a good amount of development time. We do have hopes to include it in the future, but for the moment it is not a focus of development.

If you're having problems rendering an image due to crashes, power outages, etc. I suggest trying to render in crops, which will finish sooner and thus save your "progress". You can then stitch the crops together later, though this is of course an added step. It's the best solution for now...

- Oshyan

[commorancy]

This is actually a very commonly requested feature. Unfortunately it's not nearly as simple to implement as it might seem. As a matter of fact the majority of render engines don't have this feature, even high-end professional systems like Vray or Mental Ray. There are of course some renderers that do allow this, but it would require a good amount of development time. We do have hopes to include it in the future, but for the moment it is not a focus of development.

If you're having problems rendering an image due to crashes, power outages, etc. I suggest trying to render in crops, which will finish sooner and thus save your "progress". You can then stitch the crops together later, though this is of course an added step. It's the best solution for now...

- Oshyan

Thanks for letting us know you are considering it.  I understand that it takes some programming effort to rebuild what's in memory easily and quickly restart the render.  But, with disk space amounts being readily available, the easy approach would be to create a scratch disk and write the rendered data to the scratch disk at periodic intervals.  This at least gets you back to the image where you were (and possibly even the rendering engine's most recent state).  Even if you only write the scratch disk every 15 minutes, that means you only lose at most 15 minutes of render time.  That's far better than 48 hours lost.

I also understand that most renderers don't offer this option, but that's because they offer other compensation for this feature lack.  However, just because other rendering engines doesn't offer it doesn't mean it isn't a valid request.  Large commercial apps, however, replace that feature with network rendering.  So, Vray and Mental Ray and even Carrara offer network rendering processes to compensate for the lack of resume.  So, while you cannot easily resume one machine that might crash, you may still have 4 other systems rendering so the work isn't 100% lost.  Basically, I'm talking about rendundancy vs resuming.  Since TG2 doesn't offer any redundancy options (such as network rendering), the only other option is to go with a resume feature.  Of course, you don't need to go with a resume feature if you decide to add network rendering (which may be the better option).  The only reason I suggest resume over network rendering is the amount of synchronization necessary to get all of the network renderers on the same page.  Resume only requires one machine to be synchronized.  In other words, I would think implementing network rendering would be far more difficult than writing a scratch disk with some internal data structures for resume.

As far as crops, I've been there and done that.  My problem with rendering crops is that there is still an issue with mismatch coloration on crops.  It seems that the GI changes for each calculated crop section, so the overall coloration of individual crops don't match.  When you stitch them together, the colors don't always match up and that requires a lot of time doing color correction matching on each crop in postwork.  On top of that, since color matching isn't always perfect, it requires additional hand clone-tool touch up in the Gimp.  For right now, rendering the image all at once saves that time with postwork.

Thanks.

--
Brian

[neuspadrin]

As far as crops, I've been there and done that.  My problem with rendering crops is that there is still an issue with mismatch coloration on crops.  It seems that the GI changes for each calculated crop section, so the overall coloration of individual crops don't match.  When you stitch them together, the colors don't always match up and that requires a lot of time doing color correction matching on each crop in postwork.  On top of that, since color matching isn't always perfect, it requires additional hand clone-tool touch up in the Gimp.  For right now, rendering the image all at once saves that time with post work.

Have you tried using the ray detail region setting to change it out of the default (cropped region only).  This  usually increases the accuracy if you change it to camera. Its on the advanced tab for the renderer.  It will add a little time to cropped renders but definitely makes the stitching easier.

Also I'm sure planetside isn't just ignoring it or anything, its just right now a few other features take priority (64bit edition for example).  And one thing to keep in mind is planetside has 2 programmers to implement these features so they tackle issues as they can.

[Henry Blewer]

I am looking forward to this being included. For me, even a reasonable size like 2400 x 1200 is a two day render. (at least)

[Oshyan]

I'm not sure myself, but my guess would be network rendering may actually be easier than render resume. But don't quote me on that. Fortunately it also has greater utility, so quite honestly I think we'll see that before resume features.

As for crop render issues, yes there are still problems with GI. Higher GI Sample Quality and GI Blur Radius can help with this however.

- Oshyan

[Matt]

A basic, approximate resume feature wouldn't be difficult to implement, as long as you're OK with it restarting whole buckets (tiles) that weren't completed last time and with populations etc. needing to be recalculated before resuming. If that's the goal, then essentially all you need to store is the project file, the image rendered so far, and a record of which buckets finished rendering already.

Matt

[Oshyan]

Thanks for the details Matt. So far we don't have GI caching, so that too would need to be implemented, or the GI pre-pass would have to be calculated when resuming to avoid the GI mismatch issues that happen with crops. So it's definitely possible, but would still take some time to implement properly. As we've discussed, this is still a couple months of work.

- Oshyan

[Henry Blewer]

It would be a fantastic feature. Incendia has this, but Incendia uses much less data, and calculates the image differently.