GIC gathering

Started by FlynnAD, September 14, 2013, 09:12:39 PM

Previous topic - Next topic

FlynnAD

Hi Planetside,

Say you pre-calc the GI cache for an animation, so you've got a sparse set of .%04d.gic files. Then, when you're ready to render the beauty scenes, you change the GI caching to Read Only, and choose any one of the .gic files (if I am doing this correctly).

If then you gather the project, the suite of gic files are not gathered; only the specific one you noted is gathered. This causes a bit of headache when transferring the scene to another computer for network (Backburner or other) rendering.

Right now I'm working around it by using the network path to the single Read Only gic file. I think this works, but it's hard to tell for sure, because the slave computers don't give any feedback (whether they do find the suite of gic files or else can't find them and move on). I could probably manually copy and paste all the gic files from wherever I saved them into the gathered-project-folder.

Is there a way to auto-gather all the gic files? If not, could there be a choice whether to include or not the suite of gic files, within the Gather Project dialog box?

Thanks,
Matt

Oshyan

File sequences in general are not included in project gathering at this time (other types would be object or image sequences, for example). I'd like to have that option in the future, it should be possible. But it would probably just create 1 gathered project, not one for each of a given set of frames for example. So it's really no different than just copying all the GIC files across to each location, which is what we recommend *if* you don't want to use a network drive. Basing it all on a network drive, both project file reading, and image file writing, is the most consistent and smooth way I think.

- Oshyan

FlynnAD

Doh! (Homer Simpson-style, along with a forehead slap):

Oshyan, I agreed with you, and went with the network .gic path rather than copying all the .gic files into the Gathered Project Assets folder (this method seems like a bloat of disk space). Then rendered an animation, and saw afterwards that none of the indirect illumination worked.

I found out that upon gathering the project, even if you have a network path noted in the Read GIC file dialog box, the gathering process will overwrite this with the location of the single .gic file that you've specified, which TG3 has saved into the Gathered Project Assets folder.

So either you have to manually go back into the gathered .tgd file and change the Read GIC location, or copy all the gic files over.

This is not a bug, but it is a user-headache. I'm wondering how many other people it'll snag. Can the Read GIC dialog box be programmed to save a network path, if a network path is written in? TG3 does save the network paths for the render output/ extra images output and doesn't change them upon gathering.

Thanks,
Matt

Oshyan

I'm confused. Why are you specifying a single GIC file in the first place? There may still be a bug here, but it sounds like you're doing something odd (or at least something I wouldn't have done) which may be complicating things. Does the Gather process overwrite the GIC file path if you A: specify a network path name and B: make it a file sequence, rather than single file (e.g. gicache_%04d.gic)? The whole point of gathering is to make all paths local/relative, so it's not surprising it would overwrite a network path, it overwrites *all* paths (that are not already relative).

- Oshyan

FlynnAD

Oshyan,

From
http://www.planetside.co.uk/wiki/index.php?title=Terragen_2_Global_Illumination:

"Rendering a final animation

    Switch to "Read GI cache file(s)" in Render GI Settings. Specify the filename of one of the GI cache files that you generated. Any file in the sequence will do. "

After reading this on the PS Wiki, yes, I was picking a specific .gic file. I was also naturally picking a specific .gic file because if you click on the folder icon to the right of the GIC cache file location text slot, you scroll to the network drive and folder you want and then click on a specific .gic file. But WITHIN that chooser dialog box, you cannot replace the gic frame name with %04d. That dialog box will tell you that the " '...%04d.gic' file is not found". So you have to choose a specific .gic frame file, click ok, and then back within the main TG3 GI render settings node box, replace the frame number with %04d.gic. If you do this, then yes, you're right that it does work for network drives, thanks. Finally got it.

Is there a way that TG3 could have an option within the dialog box when you go to choose the location of the .gic files, to check a box that says "GIC sequence"? This would be very similar to AfterEffects; when you import a file, at the bottom of the AE import box, there is a checkbox marked "EXR Sequence" where you can tell AE that yes, you do mean to choose a sequence, or no, you really only want the one single file. In TG3, checking this box, even though you've picked a specific frame gic file, would tell TG3 to replace that "network/folder/file.001.gic" to "network/folder/file.%04d.gic" and avoid errors.

Thanks for the help,
Matt

Oshyan

Good feedback. Yes, I had forgotten that when doing file picking, it does require you to select a single file. I agree that having a sequence option would be good. We'll look into it.

- Oshyan

DutchDimension

Hi there,

thread resurrection.  :o
Has this bug where just one GIC file is gathered (instead of the whole sequence) been fixed in the latest version? It is still very much present in build 4.1.18 which we're currently running.

I didn't see any mention of it in the change log.

Also, could you please for the love of our sanity make sure the file browser windows remember their latest position? Having to re-navigate 10+ layer deep hierarchies in a big company network every single time you want to save/load/assign location, is never an option a modern day app should impose upon its user. It's 2018 for heaven's sake.  :)
In this case it's the 'Choose folder' button in the 'Gather Project' window, but really this should apply to all file browser windows.

Cheers!

Dune

QuoteAlso, could you please for the love of our sanity make sure the file browser windows remember their latest position?
`I think it does. If I add an object, browse to my object folders, etc, then want to add a clip, browsing starts with the clips folder.
In fact, I find it annoying sometimes if I alternatively load objects, or clips, or image maps.... and a shortlist of latest (5 or so) folders would be good to have a choice from. If feasible.

Oshyan

File sequences of any kind are not supported by the Gather function at this point. It's something we'll need to add in the future.

- Oshyan

DutchDimension

Quote from: Oshyan on August 06, 2018, 06:59:20 PM
File sequences of any kind are not supported by the Gather function at this point. It's something we'll need to add in the future.

- Oshyan

I'll say. Can't believe this is still M.I.A. after 5 years since the original post flagged it.

It's rather fundamental to attempt any sort of pipeline integration of TG. Coming back to the office on Monday morning and discovering your weekend render is useless because your caches are incomplete on the farm is a big bloody no-no.

DutchDimension

Quote from: Dune on August 06, 2018, 01:23:59 AM
QuoteAlso, could you please for the love of our sanity make sure the file browser windows remember their latest position?
`I think it does. If I add an object, browse to my object folders, etc, then want to add a clip, browsing starts with the clips folder.
In fact, I find it annoying sometimes if I alternatively load objects, or clips, or image maps.... and a shortlist of latest (5 or so) folders would be good to have a choice from. If feasible.

I like how it's handled in Maya. ie. Create a new project and Maya creates the necessary folder structure (completely customizable of course), then any file dialog jumps to the root of said project and remembers it's position. Maya native browser also has a handy index of sub folder shortcuts for quick access. TG should have something like this by now.

digitalguru

#11
QuoteTG should have something like this by now.

That's a great idea, that's something I do by default whenever I set up a new Terragen scene - I copy a set of folder templates I've made:

Of course, if I then have to gather the project, this organisation goes out of the window :-)

[attachimg=1]


ajcgi

#12
Quote from: DutchDimension on August 06, 2018, 07:38:45 PM
Quote from: Oshyan on August 06, 2018, 06:59:20 PM
File sequences of any kind are not supported by the Gather function at this point. It's something we'll need to add in the future.

- Oshyan

I'll say. Can't believe this is still M.I.A. after 5 years since the original post flagged it.

It's rather fundamental to attempt any sort of pipeline integration of TG. Coming back to the office on Monday morning and discovering your weekend render is useless because your caches are incomplete on the farm is a big bloody no-no.

[edit] Just realised I answered the wrong question here, but while I'm writing... caches being gathered would be nice, but aren't you storing them on a network path already? That's how we approach things here. And yes we also have a hierarchy of useful folders that are copied into every new shot. ;)

jwiede

#13
Quote from: Oshyan on August 06, 2018, 06:59:20 PM
File sequences of any kind are not supported by the Gather function at this point. It's something we'll need to add in the future.

I think you're kind of missing the point in this case:  The real problem isn't that the file sequence isn't being gathered.  The real problem is that there's no way to tell the gather process to just point the (newly-produced) "gathered scene" to a cited path, rather than have it attempt to gather the single file referenced by that path. 

Put another way, it's about telling the gather process whether to gather or just keep as a path reference.  There are many cases where if you've got shared content stored on network shares or such, you might prefer to keep those as "common" path references rather than gathering them (read as: duplicating them) into the package.

That the gather mechanism can't gather file sequences is secondary.  Even if it could, the user already has the file sequence on a network share, so wouldn't necessarily want it to be gathered.  It's about allowing the user to decide what gets gathered and what gets referenced on a finer-grain level than just by-scene.  Esp. in group work environments, having the ability to gather scenes while retaining shared content references (instead of copying them) is quite useful. 

Another case where gathering is often undesirable involves certain types of easily-regenerable cache data:  In many cases, it is often more efficient to have the recipient regenerate the cache data on their end, rather than for the sender to package and transport large quantities of cache data.  Having the ability to tell the gather mechanism to just reference the path rather than gather the file(s) in such cases can help minimize distribution archive sizes.

Oshyan

Understood, thanks for those additional details. I've added them as an enhancement request for the Gather function in our project tracker.

- Oshyan