Using Terragen in production environments

[gregtee]

Hi all,

I'm relatively new to the Terragen community though I've used Terragen since its Classic release and an early pre 2.0 version when it was in development as a Linux based app at Digital Domain for the show Stealth, though I haven't really used it much since then, though its matured a lot. 

Recently I started using it again (2.5) for some other projects at DD and have some thoughts/comments about how it functions as production software and wanted to know what others who use it in this capacity think about its functionality. 

Overall the software performed quite well and did what I needed it to do, though the main things that hampered productivity were as follows:

First, I really missed the Linux version. As a facility we still have the old pre 2.0 Terragen with early versions of some of what are now standard features, but it would be great to have a current version that runs on Linux so that artists don't have to have dual workstation setups at their desks. 

Next, the part of the software that most seemed to be in need of further development was its animation and scene management tools, along with its integration with other software platforms such as Maya and Houdini. 

With regards to animation, the biggest issue I had was with its lack of parenting abilities and very basic curve editor.  Not being able to quickly move animation from one source to another in a single file is a pain. I know the chan file exists but not every animateable paremeter supports it.  For example if I want to take the animation of a rotating planet and put it on a texture transform node, the only way to do it it seems is to hand copy, one at a time, the curves from each axis of the source and paste them into the corresponding rotational axis of the target node.  You also have to make sure you're doing this with frame 0 set on the timeline or your animation will be offsett however many frames your timeline was set at.

Lack of parenting is another problem.  For example, my scene required that I have a rotating planet with clouds attached to the atmosphere.  Since you can't attach the atmosphere to the planet as it moves, the only way to get the clouds to stick to their positions as the planet rotated was to track a projection camera that stayed stationary over its respective place on the earth as the earth turned. Spherical mapping was not an option btw as the resolution needed for the cloud's image map wasn't of sufficient resolution. I figured I'd need a map of at least 85k across to maintain map detail and TG would crash at anything over about 50k, so projection cameras with high res satellite images were used instead.   This meant we had to export a camera from Maya what was constrained to a rotating planet and written out as an FBX file, and while this worked quit well, it was a more time consuming and cumbersome process to go through then normally one would want to in an environment with tight deadlines.  An easier solution would have been to just position a projection camera where you need it, target it to the planet's center so your projection is square on the surface of the atmosphere and then lock it into position so no matter how the planet rotates the projection camera always stays locked to its position.  This would have other uses too for projecting terrain maps and such.

The overall interface with regards to how TG displays its world with the constantly subdividing mesh tends to slow workflow speed.  It would be nice if instead of trying to constantly render polygons every time you touch something in the interface that it would just display a wireframe of everything and not try and constantly subdivide a sort of rendered half resolved mesh.  This is something that Vue does quite well and it drove some of my artists a little crazy because the machines they were working on weren't the fastest ones we have, so navigating around their scenes was often times cumbersome.  Wireframe options that don't try and constantly resolve themselves into higher levels of detail would be preferable.

It would be nice, from an integration with other software packages to have Deepshadow support. This would allow TG to export all of its rendered subdivided detail to Houdini, allowing for more artists to attack different aspects of a shot, leaving TG artists to do what TG does best, and offloading things it doesn't do as well, such as procedural animation such as moving rivers and waterfalls for example to these other packages. 

The ability to set up shot level presets would be useful.  For example, having certain render settings, camera FOVs and such pre defined and loadable into TG scenes would be useful.  It's great that the FBX loader handles this for the camera and shot length, but also having render settings pre defined and accessible depending on the needs of a shot would be helpful. 

Another small item would be atomatically hooking up nodes when you hit the N key to drop new ones in.  For example, if I have an image shader node highlighted, hit the N key and add an transform shader node, it should drop in already hooked up to the above highlited node.  I know its a small thing, but when you're working fast these little things really speed up your workflow as they eliminate lots of mouse clicks hooking things up.   Nuke does this well. 

I think the most impressive thing about TG is how it just chews through tons of displaced geo and renders very realistic environments quite quickly.  I also like its clean interface, which still reminds me a lot like early versions of Nuke before The Foundry cluttered it up. 

Python support would be huge.  Many of our facility pipeline tools use python and having hooks into TG so that other workflows can more seamlessly integrate with it would be extremely helpful.

I'd be curious to know what others think of this and what their experiences have been using TG in fast paced production environments and how they feel it could be improved. 

Thanks

-Greg

[Tangled-Universe]

Hi Greg(?)

That's quite an extensive post

It would be best if a Planetside representative would answer to your remarks and questions, but as an experienced user I might be able to clarify a couple of things, although some of your workflow descriptions are a bit difficult to follow for me.

First, I really missed the Linux version. As a facility we still have the old pre 2.0 Terragen with early versions of some of what are now standard features, but it would be great to have a current version that runs on Linux so that artists don't have to have dual workstation setups at their desks. 

I remember some people had TG2 run succesfully using Wine.
That was quite a while ago though, so I'm not sure how it will do at this moment.

With regards to animation, the biggest issue I had was with its lack of parenting abilities and very basic curve editor.  Not being able to quickly move animation from one source to another in a single file is a pain. I know the chan file exists but not every animateable paremeter supports it.  For example if I want to take the animation of a rotating planet and put it on a texture transform node, the only way to do it it seems is to hand copy, one at a time, the curves from each axis of the source and paste them into the corresponding rotational axis of the target node.  You also have to make sure you're doing this with frame 0 set on the timeline or your animation will be offsett however many frames your timeline was set at.

I suppose you did have a look at the animation curve edit panel? You can access it by hitting F8. I guess you did find and use it, but found it too limited in fuctionality?
If I'm not mistaken it allows for copying curves from one parameter to the other.

Lack of parenting is another problem.  For example, my scene required that I have a rotating planet with clouds attached to the atmosphere.  Since you can't attach the atmosphere to the planet as it moves, the only way to get the clouds to stick to their positions as the planet rotated was to track a projection camera that stayed stationary over its respective place on the earth as the earth turned. Spherical mapping was not an option btw as the resolution needed for the cloud's image map wasn't of sufficient resolution. I figured I'd need a map of at least 85k across to maintain map detail and TG would crash at anything over about 50k, so projection cameras with high res satellite images were used instead.   This meant we had to export a camera from Maya what was constrained to a rotating planet and written out as an FBX file, and while this worked quit well, it was a more time consuming and cumbersome process to go through then normally one would want to in an environment with tight deadlines.  An easier solution would have been to just position a projection camera where you need it, target it to the planet's center so your projection is square on the surface of the atmosphere and then lock it into position so no matter how the planet rotates the projection camera always stays locked to its position.  This would have other uses too for projecting terrain maps and such.

Parenting abilities, to my knowledge, is a feature the development team is aware of as it has been discussed here before.

As far as rotating clouds/textures and the like in 2.5. I think that should work now that some shaders, like the transform shader and planet shader have additional texture controls.
I don't know it by heart at this moment at work, but I can have a look at it tonight.

The overall interface with regards to how TG displays its world with the constantly subdividing mesh tends to slow workflow speed.  It would be nice if instead of trying to constantly render polygons every time you touch something in the interface that it would just display a wireframe of everything and not try and constantly subdivide a sort of rendered half resolved mesh.  This is something that Vue does quite well and it drove some of my artists a little crazy because the machines they were working on weren't the fastest ones we have, so navigating around their scenes was often times cumbersome.  Wireframe options that don't try and constantly resolve themselves into higher levels of detail would be preferable.

I'm not familiar with how Vue is doing this and if it is doing this whether it is with a fully procedural planet?
As far as I can recall Vue can't handle such scenes, still.

Anyway, regardless of that, I think a wireframe option would be nice.
Further you can of course just pause the preview make the adjustments and unpause to see what it does.
A wireframe preview which does not subdivide into further detail would mean that for the entire camera view TG would need to cache geometry, which I can imagine in many situations isn't feasible.
If you would have the terrain shown as wireframe then we would also need a slider for the amount of subdivisions applied in the 3D preview.

I'm a bit afraid that there's also the possibility of a user error here.
The 3D preview render speed relies quite heavily on your atmosphere complexity and the atmosphere render settings.
Just let a 3D preview finish with 8 and 128 atmosphere samples and you'll see how it affects the 3D preview. The same accounts for clouds.

In many situations, if you are just working on the terrain for example, it's just best to switch off the rendering of atmosphere for the 3D preview.

The ability to set up shot level presets would be useful.  For example, having certain render settings, camera FOVs and such pre defined and loadable into TG scenes would be useful.  It's great that the FBX loader handles this for the camera and shot length, but also having render settings pre defined and accessible depending on the needs of a shot would be helpful. 

There are 2 options for this:
1) clip files
2) customised default scene

1) as you may know you can save a selection of nodes as a clipfile and import that into another TG project.
This is useful if you have, for example, a solid working terrain texturing network which you want to apply into other projects.
Select the entire shading/texturing network and save it as clipfile. Saves you a lot of time

2) If you use certain mini-networks (function based voronoi noise networks) or always the same test render settings then it would save you a lot of time to customise the default scene for TG2.
You can adjust renderers, cameras and what not to your preferences so that they are always available when you start up TG2.

or

A nice feature would be to have the "node palette" contain a clipfile library which you can drag and drop into your network.

Python support would be huge.  Many of our facility pipeline tools use python and having hooks into TG so that other workflows can more seamlessly integrate with it would be extremely helpful.

Python is on the developments' team radar:
http://www.planetside.co.uk/forums/index.php?topic=12043.msg121368#msg121368
http://www.planetside.co.uk/forums/index.php?topic=7097.msg75528#msg75528
http://www.planetside.co.uk/forums/index.php?topic=7132.msg77774#msg77774

To summarize I think a lot of the experiencies you have layed out here are not unique or new so to say.
It's safe to say that the majority is also known to the Planetside team.
However, like I stated before it might be best if one of them chimes in here to give you the most up to date information and inspiration.

Cheers,
Martin

[penboack]

First, I really missed the Linux version. As a facility we still have the old pre 2.0 Terragen with early versions of some of what are now standard features, but it would be great to have a current version that runs on Linux so that artists don't have to have dual workstation setups at their desks.

Have you though of using Terragen via remote desktop rather than having a second workstation on the desk?

The overall interface with regards to how TG displays its world with the constantly subdividing mesh tends to slow workflow speed.  It would be nice if instead of trying to constantly render polygons every time you touch something in the interface that it would just display a wireframe of everything and not try and constantly subdivide a sort of rendered half resolved mesh.  This is something that Vue does quite well and it drove some of my artists a little crazy because the machines they were working on weren't the fastest ones we have, so navigating around their scenes was often times cumbersome.  Wireframe options that don't try and constantly resolve themselves into higher levels of detail would be preferable.

Regarding you comments on Vue, as I use both Vue and Terragen I may as well chime in.

I find the performance of Vue degrades very quickly as scenes get larger. Try working with EcoSystems on a 10km x 10km terrain and you will quickly see what I mean!
In addition, I find that Vue is almost completely unusable if you are working with Spherical Planets firstly because the Viewport arrangement is completely unsuitable and secondly because render times are all too frequently a multiple of what they are without the spherical planet option enabled.


There is one area, that you don't mention, which is the lack of render layers/passes in Terragen. As well as speeding up the workflow, by allowing much more to be done in post rather than adjusted pre-render, in many situations I think that they would make working with other applications much easier. I believe that these may have been promised for a future release.

[Tangled-Universe]

Ah so I wasn't mistaken by Vue's incapability of dealing with large scale scenes

[gregtee]

There is one area, that you don't mention, which is the lack of render layers/passes in Terragen. As well as speeding up the workflow, by allowing much more to be done in post rather than adjusted pre-render, in many situations I think that they would make working with other applications much easier. I believe that these may have been promised for a future release.

I believe you're probably right. 

Even with Remote Desktop you'd still need two workstations, one running windows and the other sunning Linux right?  That's the configuration I'm trying to avoid.  I already have shared monitors and keyboards through switch nodes. 

[Tangled-Universe]

For 3D preview surface rendering it might be interesting for Planetside to have a look at the new Opensubdiv standard from Pixar.
(Actually it might be interesting anyway so that TG is using an industry standardized format for exchanging geometry as this goes beyond FBX)

http://graphics.pixar.com/opensubdiv/index.html
http://graphics.pixar.com/opensubdiv/architecture.html
Based on:
http://research.microsoft.com/en-us/um/people/cloop/tog2012.pdf

[zaxxon]

The rise of GPU technologies along with emerging standards like Ptex and now with Pixar's most generous release of their Subdiv libraries into the public domain augers well for 3D artists. Be great if Planetside could incorporate some of those tools.

[Tangled-Universe]

OpenSubdiv is also suitable for CPU.

[jo]

Hi Martin,

For 3D preview surface rendering it might be interesting for Planetside to have a look at the new Opensubdiv standard from Pixar.
(Actually it might be interesting anyway so that TG is using an industry standardized format for exchanging geometry as this goes beyond FBX)

This isn't really relevant to TG in the way you might think it is. Although TG does work by subdividing and displacing geometry it's a different thing than the the actual subdivision surfaces that OpenSubdiv is used for. If you look at your second link you can see a bit of a description of what subdivision surfaces are actually used for and if you search for "subdivision surface" you'll find much more of course :-).

Regards,

Jo

[jo]

Hi Greg,

Thanks for the detailed feedback. I would say we're aware of all the issues you brought up and for some of them improvements are in the pipeline. Martin did a good job of addressing your points, thanks Martin :-). I will just add a bit to a couple of them.

Regarding the animation curve editor, it is pretty basic at the moment. We do have further improvements planned. I especially take note of what you say about moving animation data between parameters.

I would like to emphasise what Martin said about clip files. This is the way to quickly add presets into scenes.

Of course we've very interested in any other feedback you have, as well what others have to say.

Regards,

Jo

[Tangled-Universe]

Hi Martin,

For 3D preview surface rendering it might be interesting for Planetside to have a look at the new Opensubdiv standard from Pixar.
(Actually it might be interesting anyway so that TG is using an industry standardized format for exchanging geometry as this goes beyond FBX)

This isn't really relevant to TG in the way you might think it is. Although TG does work by subdividing and displacing geometry it's a different thing than the the actual subdivision surfaces that OpenSubdiv is used for. If you look at your second link you can see a bit of a description of what subdivision surfaces are actually used for and if you search for "subdivision surface" you'll find much more of course :-).

Regards,

Jo

Thanks Jo,

Is it?
It describes that Opensubdiv is useful for drawing geometry, not rendering it, so it definitely is useful if you consider wireframe preview rendering.
Perhaps I should have been more specific about wireframe rather than preview rendering.

Since you mention that Opensubdiv subdivisions differ from TG, how is that?
I can see that TG possibly might use a different algorithm for applying subdivisions on primitives, but subdivisions are subdivisions?

Martin

[jo]

Hi Martin,

Subdivision surfaces are a special type of surface. TG doesn't use them. I have a headache right now and I'm not feeling up to explaining it so I would suggest doing a search for "subdivision surface" and going from there :-).

Regards,

Jo

[Tangled-Universe]

Ghehe well sorry for causing you headaches

It's an eye opener for me that TG doesn't use subdivision surfaces because I never knew any better from references* and discussions here than that the detail slider in the renderer is a subdivision controlling parameter.
* http://magnuswrenninge.com/wp-content/uploads/2010/03/Fairclough_Wrenninge-RobustRenderingOfHighResolutionTerrain.pdf
Just to explain myself where I'm coming from with my claim.

I'll have a look at it anyway.

Thanks

[Oshyan]

Well, Martin and Jo have both done a good job of responding to most of this. I just wanted to chime in and say that it's great to get this kind of feedback from a production standpoint, and even better to see that we already have plans to address a good number of these issues and requests in upcoming versions.

In the meantime, I think getting to know TG's less well-known but very handy features, like the node clip system, custom default scene, pop-op node palette, custom shortcut keys, etc. will serve you very well.

- Oshyan

[gregtee]

That's a good point about clip files.  I hadn't thought of that but it makes sense.  I suppose one could make a clip scene that is the default start scene when you first run the program, right? 

Having spent a little time now looking at clip files, I see that they're just text files.  This is exactly how Nuke handles it's nodes, which means that TG could be configured in the same way that Nuke is, which is great.  There's tons of user defined nuke nodes out there that nothing more than gizmos that are combinations of individual nodes hooked up in specific ways to do things.  It would be great if TG had a user defined drop down menu system like Nuke does where you could store your clip files and just add them at will, assuming this feature isn't in there already of course and I just overlooked it. 

Speaking of Nuke gizmos, that would be a great feature to add.  A Nuke gizmo, for those who don't know is collection of nodes with certain user defined controls exposed in a single interface.  So one could for example, make a cloud system and turn it into a single node that had it's own sliders and inputs so that the end user only sees what's needed instead of the entire node tree.  This really cleans up scripts and makes the whole workflow much less complicated.  Anything that can be built in TG with nodes where variables are changed depending on the input parameters could be converted into a single node gizmo.  Entire libraries of rocks, clouds, terrains, or whatever could be easily compiled into these simple to use nodes.