Hybrid encoding-rendering ?

[Kadri]

I had since a while an idea in my mind but was afraid to post it here.
I searched around but could not find similar things on the net.
Maybe because i have no clue about how to search with the right keywords etc.
Anyway here it goes:

The video encoding method to use less data with still acceptable visual quality is basically using small blocks of the frame
and checking the other frames against differences and using only those differences by using motion detection etc...

My main question is couldn't that kind of approach used for standard rendering (for animation especially) too?

For example when you have a camera move over the sky the ground changes slightly but the sky is mostly the same.
But still Terragen and the other render engines i know render the whole image as they are seeing the frame the first time all the time.

A video encoding kind of approach coupled with the fact that render engines already knows everything about the scene (motion vectors etc.)
could predict that for the next frame that the sky (for example) doesn't need to be rendered ones again and could use the data from the frame before and render only the parts that does have changes in it.

The render engine could render every 5'th frame (for example) and interpolate (Like the GI cache kinda does maybe?)
the other frames with the data it does have, render only the different parts etc. .

This would be very fast for scenes with less motion and lesser so with fast ones of course.
The quality of the render could be adjustable like in video encoding.
At least for quick previews and hopefully for final animations it would be a great help i suppose if this is doable and not already used anywhere...

It is most probably already in use in some render engine but as i said i couldn't find anything.

[Hannes]

Sounds interesting. I know nothing about programming, so I'd be interested to hear Matt's opinion.

[Kadri]

Sounds interesting. I know nothing about programming, so I'd be interested to hear Matt's opinion.

As i don't know about programming too i am curious too what he would say.

[Dune]

Smart idea. Would be terrific if Matt could make this work.

[KlausK]

hi Kadri
I am no programmer by all means myself so I`d be gladly proven wrong here.

Quote:
"The video encoding method to use less data with still acceptable visual quality is basically using small blocks of the frame
and checking the other frames against differences and using only those differences by using motion detection etc...
"My main question is couldn't that kind of approach used for standard rendering (for animation especially) too?"
End Quote.

Oh that would be in fact a cool thing...
But since encoding deals with pre-rendered images I don`t think that approach would work for turning
the polygon / wireframe representation of your 3d scene into a 2d image.

Quote:
"For example when you have a camera move over the sky the ground changes slightly but the sky is mostly the same.
But still Terragen and the other render engines i know render the whole image as they are seeing the frame the first time all the time."
End Quote.

That`s because the renderer is "seeing" / calculating the next image in the timeline for the very first time?

Quote:
"A video encoding kind of approach coupled with the fact that render engines already knows everything about the scene (motion vectors etc.)
could predict that for the next frame that the sky (for example) doesn't need to be rendered ones again and
could use the data from the frame before and render only the parts that does have changes in it."
End Quote.

As far as I know the render engine does not know anything about the frame after the one it has just rendered.
The fact that it might be able to look at the transformation values of the camera (like in your example) from start
to end frame would not help it to predict the lighting situations of all the polygons involved.
And I think that is what makes the rendering process such a time consuming thing:
sampling the scene and calculating pixel values to produce a 2d image (or something like that anyways).
I guess the only thing to speed up the rendering process is lowering the quality settings of the output in general.
Which is what you do with your render settings. Or which is what game engines do through heavy optimization
because the outcome of the image is much more predictable than the next frame in your 3d animation.
Or through clever algorythms for specific calculations. But the fundamental difference between the two processes
you want to marry is the fact that the video encoder has all the picture data available from start to finish and
the 3d render engine has not. Just my 2 cents.
As I said: gladly proven wrong (by Matt or anyone in the know).

cheers, Klaus

[Dune]

That sounds logical, unless the data from the first render can be stored temporarily, like the GI cache, and used for the second, and only dumped after a while. But then, when would that be? The renderer probably can't decide from the data stored how much a next render will be different.

[Kadri]

Klaus those are things that are rightly looking like that such an approach isn't so easy and maybe even not quite so useful for rendering.

But the main difference is that the render engine should look at the scene more from an temporal aspect then spatial (as it is used now i think).
Then you add to that an image manipulating (you really only change the pixels (kinda morphing) rather then rerendering them by using motion estimation) by using some prerendered frames (every 5'th frame for example) like keyframes in an encoder.

I can even see in my mind a renderfarm that really communicates between all the computers
that are rendering a frame but are just computing the differences of the scene.
Think about like a frame slowly gets more clearer now like in the preview window.
Now think about the scene as a whole (or how much it is feasible)
and how all the computers getting the full animation clearer rather then one frame only.

Hard to describe it and not sure if that makes sense what i wrote.

Basically the way scenes are rendered now does look from an temporal aspect very wasteful to me.

It might be useful for one computer too but should be especially with a renderfarm or cloud computing.
So far i know, now every computer after it gets the objects it needs to render the frame behaves like an isolated island at rendering.
This shouldn't be so.
Even with the overhead that will happen with computers communicating between them and just searching for differences between frames i think there is a possibility that you will get quite a boost in rendering. 
You will need quite a different render engine maybe that uses an image editor too for example.

The problem is i don't know if this would work from an programming aspect or maybe if this a thing that was tried, not worked etc...
Curious...

[Kadri]

Funny after my own post i found by using the words temporal and spatial, very technical papers a little closer for what i mean by searching.
Not quite the same i think (too much technical for me) , but closer: Spatiotemporal rendering, Video based rendering etc.