190h05m and still rendering

Started by TheBadger, February 28, 2012, 09:53:25 PM

Previous topic - Next topic

TheBadger

 :'(

This crop by its self has now taken more time than the rest of the image combined. Same settings as page 1 of this thread. I am at a total loss to understand it.
It has been eaten.

JimB

Can you render the painted shader as a greyscale mask, then reproject from the camera the greyscale render onto the scene to use as an alternative to the painted shader mask?
Some bits and bobs
The Galileo Fallacy, 'Argumentum ad Galileus':
"They laughed at Galileo. They're laughing at me. Therefore I am the next Galileo."

Nope. Galileo was right for the simpler reason that he was right.

ajcgi

I occasionally find that when I render crops they take longer than a larger crop, especially when there's lots of tiny details in there. I can only presume it's something to do with memory allocations.

TheBadger

#93
Quote from: JimB on March 21, 2012, 07:26:42 AM
Can you render the painted shader as a greyscale mask, then reproject from the camera the greyscale render onto the scene to use as an alternative to the painted shader mask?

Hi JimB. Its very cool of you to try and help me despite our differences.

To answer your question, yes, maybe. But I think that I would just not try it at all. The painted shader is not really even doing much. And what I was trying to do with it, I can just do in post. So at this point... :-\


@ajcgi
Thanks. But I just don't know. Hopefully when Oshyan returns, he will be able to understand this better.



Update
The crop just finished. All of the black space you see in this image below rendered in like 30-40 hours. The crop took 265hrs. Its that bottom left corner. I did crops of the same size in other places of the image, the average time was 6 hours.

It has been eaten.

TheBadger

The problem as I can see it, is that the renderer gets exponentially slower the closer it gets to finishing. My computer is using a consistent amount of memory but not a consistent amount of threads.
It has been eaten.

rcallicotte

I still think this TG2 renderer is one of the best quality in CG.  
So this is Disney World.  Can we live here?

dandelO

I'd definitely use Jim's advice, TB. Using painted shaders really slows things down, I always look for another option before deciding to use the painted shader, for this very reason.
The more strokes you use, the worse it becomes. Using an image of your painted mask would really speed this up I'd think. Painted shaders are good in theory but there are some really bad issues with render time.

FrankB

Quote from: dandelO on March 22, 2012, 02:58:46 PM
I'd definitely use Jim's advice, TB. Using painted shaders really slows things down, I always look for another option before deciding to use the painted shader, for this very reason.
The more strokes you use, the worse it becomes. Using an image of your painted mask would really speed this up I'd think. Painted shaders are good in theory but there are some really bad issues with render time.

totally agree. Just make sure the rendered mask image is really high resolution.

TheBadger

#98
Hey guys, thanks. I always listen to your advise, believe me!

Oshyan,

If you have time, when you look at this project file. Would you please try rendering a view from ground level/6ft, from somewhere facing from a different direction then my current camera?! When I render with all objects turned off-no problem. When I render with objects turned on- everything in the image view warps. Walls end up right in front of the camera (they move) really strange stuff. this is even a bigger problem than render time, believe it or not.

Ready to quit on this. One problem after another. Please help.
It has been eaten.

Oshyan

Badger, can you just send me a TGD file (without objects, of course) that reproduces the issue? Actually, even better, just copy and paste a camera node with the right position into the forums here. You can do this easily by highlighting (clicking on) the camera node, using Copy (Ctrl-C), and then go to the forum post window and Ctrl-V/Paste. You'll see your camera node clip file show up as text and I can easily drop that into my version of the scene to test this.

As for the overall testing, figuring out what's going on, I'm not really sure how much further I can go with it. RAM seems to be a limitation for me at this point. Due to the amount of objects with duplicate textures you have, memory use is really much higher than it needs to be (largely a TG limitation). It would take ages to fix this in the node network though.

Essentially I think your rendering problems are down to a combination of wanting detailed lighting (thus high GI settings), and having lots of objects/memory use. Now that I'm back from vacation I can look into it more if you like, it's possible I will come across some eureka realization, but I kind of doubt it as I already skimmed through the most common problem settings. It's just a beast of a scene for TG to render. Frankly you may have better luck doing the maze in another program and compositing in the sky/background/lighting.

- Oshyan

jo

I've made a version of the file which consolidates all the ivy textures (text editor and regex :-). There are still some wrinkles to sort out but I'll send it to you when I'm done Oshyan. I also took out the painted shaders because I think Badger said they weren't really needed.

Badger, would it be a problem if the bulk of the maze was only made up of the edge maze unit? That way you could just use an image 1/9th the size. You could still stick the centre part in the middle of the scene. I'm not really sure how much memory that would save but it would be a few hundred MB at least. It's a bit of a drop in the bucket compared to the whole scene though!

Regards,

Jo

Dune


jo

Hi Ulco,

Regex is short for "regular expression". Basically it's kind of an advanced type of text searching. My text editor lets you use regular expressions for find and replace. I opened the project in the text editor. Basically what I did was copy and paste the default shaders from one object and then change all the other objects which used those shaders so they all linked to the ones I copied.

I haven't done much with regular expressions before so it took a bit of figuring out.

Regards,

Jo

TheBadger

@Oshyan,

Regarding that other issue- The problem seems to have been resolved by a restart. Thanks for wanting to help, Im just glad it turned out to be nothing.

On the issue of repeating textures... Is there a thread that talks about how I may set this up correctly in the first place? You have said that fixing it now would be a ton of work, implying that I may have been able to do it from the beginning. If so, how?

You asked if I want you to keep trying to figure this project out. No, you have done enough for me. Thank you. I think its clear that I just went about everything in the wrong way. That is, trying to do it all in TG alone. I guess the only thing left I would ask about is the stuff Jo is mentioning in his last 2 posts in this thread, in relation to repeating textures.

@jo
Hi jo
I dont think there would be any problem making changes to the maze, jo. Although I'm not entirely sure of what you mean, I will live with whatever you guys conclude on this. Can you elaborate on your text editing? Is this something I should know? Does it apply to what I asked Oshyan about repeating textures. And how would you answer that same question?
Thank you jo, very much.

OK, so... I finished the renders and crops of the scene. I worked on the image in photoshop, and just have some painting to do. Basically the image is done.
When I first started this project in other threads I was told I should build the maze in a modeling program and import. Actually several people have said so since then. But I did not have the ability to do it. Now I do, and I will take that advice.
In the end, the way I went about this can't work for more than one or two shots. Its just not practical.
Hell of a try though!
I will continue on this after I have built the maze in houdini.

Thank you everyone.
It has been eaten.

jo

Hi Badger,

Quote from: TheBadger on March 26, 2012, 03:17:18 PM
On the issue of repeating textures... Is there a thread that talks about how I may set this up correctly in the first place? You have said that fixing it now would be a ton of work, implying that I may have been able to do it from the beginning. If so, how?

I don't think there is any way to set this up in the best way easily. Basically you have a bunch of models that all use the same images for textures. When TG2 loads each model it creates a set of default shaders which each load their own copy of the image. For example with 270 ivy models with 3 texture and 2 alpha images each you end up with 1350 images loaded. The ideal situation would be that only 5 images were loaded.

Unfortunately there isn't a way for you to set this up in TG2 so that only the 5 images are used, unless you did a lot of work editing nodes.

Quote
I dont think there would be any problem making changes to the maze, jo. Although I'm not entirely sure of what you mean, I will live with whatever you guys conclude on this.

Your maze is created by displacing the maze image map. The image map is basically 9 separate sections - 3 rows and 3 columns. All of the outer sections look like they're the same. The centre section is different. What I was thinking is that you could build up the maze using two images. One would be just one of the "edge" sections, the top left one for example. You would use this to create a base map that tiles across the whole scene as the maze image does now. The other image would be the centre section. You could place that in the middle of your scene so you still get the difference there.

I wonder if it would be worth using a distance shader to limit the image map to the visible part of the scene. Just far enough so you still get maze along the horizons.

Quote
Can you elaborate on your text editing?

I'm not sure what more I can say than I did already. The project file is XML and can therefore be opened in a text editor and edited. I opened the file, copied the default shaders from the first ivy models and pasted them so they were outside the objects, kind of at the top level of the file. I then used regular expressions to do a find and replace on all the shader params of the Object Part nodes so they pointed to the default shaders I copied. I used regular expressions because the shader lines weren't all the same as they had names with various numbers of "_1" appended (we need to make that better!).

I then used regular expressions to do a find and replace on all the unused default shaders. I just replaced them with blank space to delete them.

Quote
Is this something I should know?

Hard to say :-). Project files are XML and can be edited in a text editor or processed by other tools which can read XML. For example if I were more of a scripting guy I could have made these changes with a Perl or Python script etc. Project files aren't especially complex so if you're used to working with this sort of stuff they're not too difficult to modify. For example I'm comfortable editing them partly because I've done a lot of HTML editing in the past.

I don't think we'd officially support editing the files as in provide any sort of guidance how to do it but we'd answer questions if they came up I guess.

Quote
Does it apply to what I asked Oshyan about repeating textures.

Yes. I edited the file so it reduced the number of images loaded from 1300 odd to 5. I also took out the painted shaders as it seemed from what you said that it was ok to do. The original file took 9.52 GB when loaded, the edited one took "only" 6.43 GB. I will send you a copy of the edited file.

Anyway, it's an impressive scene. Do you mind if we hang on to it for testing purposes? There's a bunch of things I can think of that we could try out with it. Not so much to make it render faster for you anytime soon, but in terms of loading lots of models at once, optimising texture use, memory use during rendering, that sort of thing. Test cases like this are difficult to come up with ourselves, it's great to be able to work with "real life" projects.

Regards,

Jo