VDB workflow testing

Started by Matt, June 13, 2018, 07:18:11 AM

Previous topic - Next topic

pokoy

Great result! Yeah that emitter thing was the only way to get something usable in Arnold for me, too. In Corona I could set up a realistic density/absorption combo but still, TG looks better. It's close to impossible to get any details, it all looks very fluffy most of the time with Arnold/Corona.

KlausK

#46
Hi,
this is a LightWave 2018.3 render right out of the box.
But it is not really done according to Matts rules.
So, don`t look at it this way.
I was simply curious what would come out at all without using a 3rd party render engine which seems to be the thing to do to these days.

- the small picture shows how LW imports the fbx and the vdb file here with LWs default settings for the vdb.
- I repositioned the camera
- moved the vdb a little bit closer to the front edge of the plane
- turned of off the luminosity on the plane - I guess I wasn`t meant to do so
  (took me a while til I noticed that. Wondering, why the heck I couldn`t get shadows on the plane...doh)
- that is why the small plane is in the front right right. Forgot to take it out again.

- lightsource is a distant light pointing straight down, intensity 3.14lx, no falloff

- the vdb is loaded into a null obj set to OpenVDB Primitive type
- emission channel set to density (LW read that from the file)
- scattering channel set to density also
- absorbtion channel not used
- scattering scale on both channels set to 0.01
- step size set to 1m
- interpolator was set to quadratic

The rendering took about 45 minutes on my machine.

CHeers, Klaus
/ ASUS WS Mainboard / Dual XEON E5-2640v3 / 64GB RAM / NVIDIA GeForce GTX 1070 TI / Win7 Ultimate . . . still (||-:-||)

pokoy

#47
I finished a batch of renders from 3dsmax with Corona (v2 release candidate 4)

Density had to be set to 2.5 to match TGs output.
With GI enabled, I finally had some success to get something that looks close to TG. Corona simulates anisotropic scattering, by using values from 0.6 to 0.85 (forward scaterring) it finally looked more like clouds instead of a white or gray mass. It was surprising to see that not even 10 GI bounces were enough to get a convincing look so I left it the default limit which is 25. A render with
constant scattering direction at 0.825 is attached.

Using the vdb density channel as scattering input instead of constant and a custom gradient on the output curve I came very close to the GI render from TG but it took to long to render, I'll try to play around with the options another time.


vdb_calib_01_black_cloud_no_shadows---3dsmax_Corona---Density-2.5.jpg

[attachimg=1]


vdb_calib_01_shadow_of_black_cloud---3dsmax_Corona---Density-2.5.jpg

[attachimg=2]


vdb_calib_01_white_cloud_with_gi---3dsmax_Corona---Density-2.5_ScatterDir-0.825.jpg

[attachimg=3]


For comaprison - with isotropic scattering direction (uniform light scattering in all directions) the volume looks all white:
vdb_calib_01_white_cloud_with_gi---3dsmax_Corona---Density-2.5_ScatterDir-0.0.jpg

[attachimg=4]


In conclusion, I don't see a reason why the density exported from TG should be changed, we all seem to get good results with it and looking at how differently the density is interpreted depending on the renderer it's hard to say whether the density from TG is off or not.

Matt, would it be possible to get a vdb file with more detail?  ;D

Thanks for the interesting challenge btw, it helps to understand how different media affect scattering direction in light, very interesting stuff.

Oshyan

It would be interesting to know your render times on any tests that are trying to reproduce the TG cloud scattering.

- Oshyan

Matt

#49
Quote from: pokoy on June 21, 2018, 07:27:24 AM
FYI - there's also a velocity channel included in the file (also in the first vdb file you posted) having the same channel name.

I guess this is a 3dsmax thing because I only write one grid to the file and it's called 'density' now.

Quote
The strange thing is that the Arnold volume shader has a transparency parameter which is set to 0.3679 (0-1 space). With this value, density has to be at 0.18. The shader doesn't accept more than 2 decimal places so who knows what exact value needs to be, but it looks like a good match. Disabling transparency will result in a black cloud and shadow, so I've left this value where it was.
I thought it's meant to produce 0.5 in sRGB, but it seems to be an arbitrary value, if it's linear it should be 0,2176 to produce 0.5 in sRGB, if it's in 2.2 gamma it should be 0.7297 to produce 0.5 in linear space - no clue what the reasoning is behind this as a standard value.

0.3679 ~= e^-1

This factor makes sense considering how density is usually converted to transparency: transparency = exp(-density * distance). Technically you shouldn't need a parameter like this if you also have a density multiplier (the transparency could be hardcoded at 0.3679), but I can see how it might be more intuitive to set coloured transparency this way. I don't know why the density multiplier needs to be 0.18, but I guess it might be scaled that way for backward compatibility reasons.

Quote
Scattering/GI
This is where transparency controls everything again. If I want the cloud to look good I need to increase the value but then the shadow will be a lot lighter than in TG's example. The GI result further down is what comes out when rendered with the same shader that matches the diffuse/shadow render.

I guess it all comes down to how volumetrics work in Arnold, there's no simple way to achieve the same results for diffuse/shadow and GI.
General light intensity has to be around 1.68 to produce a close match to TG's exposure/brightness.

What scattering/albedo controls does it have?

Matt
Just because milk is white doesn't mean that clouds are made of milk.

Matt

#50
Quote from: paq on June 21, 2018, 12:10:05 PM
Here's a test using the density channel as emitter (thanks to Démian from Isotropix). It's allready better than the default smoke look I had, but clouds are now self emitter :(

@Matt isn't any other possible data that you could generate in the .vdb that "we' could use to enhance the lack of inside scattering in our test ?

Quoting a reply from mdkay (Clarisse forum)

how does the shader (sic) fake the extra lightboost..

The volume shader in clarisse can mimic these phenomena with the scatter value for forward or backward transmission. But that value is static through the entire medium unless a property is provided to multiply it with.
Something like a SDF value which gives a value how close a point is to the surface of the vdb ...
Something Matt sure can add..otherwise a run with Houdini would help as well.


In the past, Terragen faked the internal scattering according to the density function, and we had "fake internal scattering" parameters to control that. But that was in Cloud Layer V2. Now in Cloud Layer V3 and Easy Cloud it isn't faked, it's calculated by scattering rays through the voxels using principles from physically based rendering, although I do cache a few things and take some shortcuts to make it faster than a properly unbiased method. A good physically based volume shader should be able to do the same, if it has a good phase function and enough scattering events (or bounces). Terragen's cloud shader is optimised to approximate hundreds of scatters within the same volume, but has a limited number of bounces between separate cloud layers (or between cloud layers and surfaces) where this optimisation isn't used (although this might be added in future).

Matt
Just because milk is white doesn't mean that clouds are made of milk.

Matt

Quote from: pokoy on June 22, 2018, 02:24:53 PM
I finished a batch of renders from 3dsmax with Corona (v2 release candidate 4)

Density had to be set to 2.5 to match TGs output.
With GI enabled, I finally had some success to get something that looks close to TG. Corona simulates anisotropic scattering, by using values from 0.6 to 0.85 (forward scaterring) it finally looked more like clouds instead of a white or gray mass. It was surprising to see that not even 10 GI bounces were enough to get a convincing look so I left it the default limit which is 25. A render with
constant scattering direction at 0.825 is attached.

This looks great. What does it look like if you increase this parameter to 0.99, or even 1.0 if that works?

Quote
Using the vdb density channel as scattering input instead of constant and a custom gradient on the output curve I came very close to the GI render from TG but it took to long to render, I'll try to play around with the options another time.

You shouldn't need to fake it like that. It looks like Corona is already doing a good job with this, and maybe it just needs to be tweaked a bit more. My TG render might be a little too dark at the edges. I just used the defaults.

Matt
Just because milk is white doesn't mean that clouds are made of milk.

Matt

#52
Quote from: Matt on June 22, 2018, 08:29:32 PM
In the past, Terragen faked the internal scattering according to the density function, and we had "fake internal scattering" parameters to control that. But that was in Cloud Layer V2. Now in Cloud Layer V3 and Easy Cloud it isn't faked, it's calculated by scattering rays through the voxels using principles from physically based rendering, although I do cache a few things and take some shortcuts to make it faster than a properly unbiased method. A good physically based volume shader should be able to do the same, if it has a good phase function and enough scattering events (or bounces). Terragen's cloud shader is optimised to approximate hundreds of scatters within the same volume, but has a limited number of bounces between separate cloud layers (or between cloud layers and surfaces) where this optimisation isn't used (although this might be added in future).

To be fair, TG's scattering isn't physically correct for a few reasons. There's a sort of transition between voxel-level scattering and scattering done with the high resolution density function. Different parts of the phase function are handled separately in order to get a high quality result fast. The ways some of these things are merged aren't exactly physically correct. The phase function used for direct visibility to the camera is based on the non-physically based paramers "sun glow amount" and "sun glow power" which gives the artist a lot of control over the silver lining and dark edges, while the higher order scattering which works at the voxel level uses a couple of extra parameters which also are only loosely physically based ("softness" and "param C"). Energy is preserved, but only approximately. How radically this deviates from a purely physically based solution probably depends on the chosen parameters, but scattering parameters are uniform throughout the volume so there's no explicit attempt to cheat the brightness in the cloud centres at all - the brightness emerges naturally due to the larger number of paths that connect the sunlight to the camera.

Matt
Just because milk is white doesn't mean that clouds are made of milk.

pokoy

#53
Thanks for the explanation, Matt, this helps to understand the differences. I guess you are also querying the noise function for some visual non-physical shortcuts, that's probably why I couldn't get the small details to be as prominent as in TG's render with Corona.

Totally missed that forward scattering has been already mentioned in the post with results from Clarisse.
I will do a render with a direction coef of 0.99 (the range goes from -0.99 to 0.99 for this parameter in Corona) but from the tests I did I recall that the outer boundaries of the cloud would render too dark since light is not scattered towards the viewer unless you happen to look at the light source 'through' the cloud.
What's interesting is that the silver lining effect is nicely visible with values from 0.5 - 0.7. Since this gives realistic results for the silver lining I think that's the range this parameter has to be set to. This also explains why TG features a separate sun glow section for clouds, it would be probably hard to control the silver lining in a realistic way if it was only density and scatter parameters.

Also, in order to get the cloud to receive any light on the bottom in the thicker areas a lot of GI bounces are needed. I can go as far as 100 in Corona (or completely unbiased) but that will take really long. My tests were done with 25 bounces, anything below that just weren't enough, and it's quite noisy too. I wonder how many bounces TG does for light scattering within the cloud.

The transparency in Arnold - interesting to see the 'scientific' reasoning behind that value. I find the controls a bit too arbitrary though but will try to get something more usable. I guess it will be very noisy and slow since scattering bounces need to be increased a lot for clouds.

ajcgi

That's all interesting info.  :D Hopefully I can take another look at this in Houdini over the next couple of days. Btw, I think the default FBX settings within Houdini probably interpreted metres as something else and went haywire with the scaling. I'll check that too once I get around to it. Need the clock cycles for TG rendering today, funnily enough.  ::)

pokoy

Matt, you asked for a render with scattering direction at maximum (0.99) from Corona, here it is:

[attachimg=1]

I can do a progression render from 0.0 to 0.99 (in 0.1 steps) when I'm back from holidays if it's helpful.

pokoy

#56
I had some free time, here's the progression from 0.0 to 0.99 for scattering direction. It's interesting how much this affects the cloud's look.

Update - I zipped the GIF file, for some reason it didn't play the animation.

pokoy

This just in: Disney released a detailed cloud VDB for research and testing purposes.

Link: https://www.disneyanimation.com/technology/datasets
PDF description of the cloud file: https://disney-animation.s3.amazonaws.com/uploads/production/data_set_asset/1/asset/Cloud_Readme.pdf

I guess it's particularly interesting for Matt in order to see how a Disney-standard cloud VDB looks like (and to have a nice visual example of the rendered result from their Hyperion renderer).

ajcgi

Quote from: pokoy on July 04, 2018, 11:34:50 AM
I had some free time, here's the progression from 0.0 to 0.99 for scattering direction. It's interesting how much this affects the cloud's look.

Is this animated? It's static on my work machine at least. Chrome and IE. Win7

pokoy

Quote from: ajcgi on July 04, 2018, 12:20:33 PM
Quote from: pokoy on July 04, 2018, 11:34:50 AM
I had some free time, here's the progression from 0.0 to 0.99 for scattering direction. It's interesting how much this affects the cloud's look.

Is this animated? It's static on my work machine at least. Chrome and IE. Win7

I just updated the original post with a zip of the GIF file, that should work.