Threads Crash and don't restart

Started by commorancy, August 01, 2008, 02:40:25 PM

Previous topic - Next topic

freelancah

Yeah I guess it could be that it runs out of memory. This doesn't happen if I render on lower resolutions, like 1280x768 and I can also bypass the problem on highres by cropping but still I feel this is a big problem to not being able to render one whole image.

freelancah

BTW I also tried with just 1 thread rendering and the same problem occured so atleast it's not related to the number of threads

commorancy

#17
Quote from: freelancah on October 02, 2008, 02:16:15 PM
BTW I also tried with just 1 thread rendering and the same problem occured so atleast it's not related to the number of threads

It doesn't seem to be related to the number of threads.  It could, however, be related to the multithreading support that was added.  I'm interested to find out if the scene renders in the previous version that didn't have multithreading enabled.  I seem to remember rendering full sized complex 3D scenes on the non-multithreaded version without these issues (obviously, very slowly).  I'm betting these renders would work on the non-multithreaded version.  So, I'm leaning towards a resource issue related to memory and/or multithreading (possibly a combination).

I'm thinking of rolling back to that version and test rendering with that version.  I'd much rather have it render completely in one pass than have to piece it all together later using tiny crops.. even if it takes several days to render.

Thanks.

--
Brian

commorancy

Quote from: freelancah on October 02, 2008, 02:16:15 PM
BTW I also tried with just 1 thread rendering and the same problem occured so atleast it's not related to the number of threads

Update: I tried rolling back to the newest non-threaded version, but that version doesn't support water subsurface transparencies (which I'm using).  There's not a shader with transparencies in that version.  So, I'm stuck using the threaded versions for this render.  You might be able to use it on your render if you're not using transparencies.

Thanks.

--
Brian

Oshyan

Brian, try setting max threads to 0. I believe this still roughly emulates the single-threaded rendering of the past.

It does seem like this may be a memory issue. The best way to tell is to monitor memory use for TG2 up until the moment it crashes. You can use Task Manager for this if you're willing to watch it (or perhaps check after it has crashed as it probably hasn't released the memory yet), or you can use more sophisticated long-term logging tools (one built-in to Windows is Perfmon, the Windows Performance Monitor, accessed through the Administrative Tools).

- Oshyan

commorancy

#20
Quote from: Oshyan on October 02, 2008, 11:41:42 PM
Brian, try setting max threads to 0. I believe this still roughly emulates the single-threaded rendering of the past.

It does seem like this may be a memory issue. The best way to tell is to monitor memory use for TG2 up until the moment it crashes. You can use Task Manager for this if you're willing to watch it (or perhaps check after it has crashed as it probably hasn't released the memory yet), or you can use more sophisticated long-term logging tools (one built-in to Windows is Perfmon, the Windows Performance Monitor, accessed through the Administrative Tools).

- Oshyan

Thanks for the tip.  I will try a render with max threads set to 0 and let you know. I sometimes see the error in the pre-render stage, but sometimes it won't happen until the final render stage begins.  I'll let you know how it goes.

I will say that Poser must have had this issue as well and resolved it.  What it looks like they did was allow the user to set the max 'bucket' size (thread render tile size).  Apparently, the smaller the bucket size, the smaller the memory footprint to render each tile.  It might be worth a try to allow the user decrease the tile size for rendering. This may then require less memory to render each tile and allow larger sized images to complete.  Right now it looks like TG2 arbitrarily picks the tile size based on some screen dimension calculation combined with GI settings.

I'm running a render with Process Explorer watching.  So far, the working memory set is up to 1,368,356K and growing. The virtual size is 2,058,296K.  It hasn't stopped yet.  Although, I will admit I reduced some GI parameters and the Detail setting in this render.  If/when it crashes, I will take a snapshot of the Process Explorer memory window and post it here.

Update: First try, it got about 3/4 through stage 1 pre-render and then I got the C++ error and the whole thing closed.  This usually happens when I don't preallocate the subdiv cache.  But, I had that set at 400.  So, not sure about why I got the C++ error.  The memory wasn't over 1.4MB working set when it crashed.  So, that didn't appear to be memory related (at least, running out).  I'm going to try the render again at my original settings. 

Thanks.

commorancy

#21
Oshyan,

After running an interesting test with Process Explorer, I'm starting to believe that the threads dying may be a thread synchronization issue.  While running Process Explorer and monitoring the memory stats for tgd.exe, the error did not occur and the render had gotten farther than it had ever gotten without crops.  I know that monitoring a process slows the process down during each data gathering iteration.  The performance monitoring obviously slows down the render threads as well, but the threads did not crash as long as Process Explorer was monitoring the process.  Out of curiosity, I quit Process Explorer during the monitor of the render (I wanted to see if it would speed up the render).  Within 5 seconds of closing Process Explorer, I had a message that a thread had died.

Note that when the thread died, the memory was still well below 2GB (around 1.2-1.3GB in the Working Set).

I'm restarting the render again and I'm planning on leaving Process Explorer running the entire time (or until an error occurs with PE running).  However, the fact that PE is slowing the process and render threads down without errors indicates something other than a memory issue.  My guess is something to do with thread synchronization.

Update: Checking this morning, 3 threads have died,  One is still working.  See the attached Memory and Messages panels.  I clicked 'Stop' in the rendering window and the smaller confirmation window tries to appear.  See EmptyPanel.jpg for example.  The contents of the cancel panel window do not draw.  While it appears that the window content is not there, in fact it is.. the content was just not drawn.  I was able to randomly click and find the button to get out of the panel.  However, note that when Window contents do not draw (like buttons and menus), this usually indicates a GDI resource issue.  I do not know if the GDI resource issue caused the threads to fail, but it's very possible since you are displaying bitmaps to the screen via the threads.

Thanks.

--
Brian

jo

Hi Brian,

It looks to me like a bit like you're running out of memory. I don't think the problem is a GDI leak because from the Process Explorer it seems like the number of GDI handles being used is about what I would expect. The reason the dialog is blank could also be down to running out of memory.

Can you please pack up all the files used in your scene and email them to me at jomeder@ihug.co.nz, or if it's getting close to 10 MB could you please upload them somewhere and send me a link? I'd like to try your scene with the Mac version and the Windows version.

Regards,

Jo

commorancy

#23
Quote from: jo on October 03, 2008, 10:39:10 PM
Hi Brian,

It looks to me like a bit like you're running out of memory. I don't think the problem is a GDI leak because from the Process Explorer it seems like the number of GDI handles being used is about what I would expect.

Can you please pack up all the files used in your scene and email them to me, or if it's getting close to 10 MB could you please upload them somewhere and send me a link? I'd like to try your scene with the Mac version and the Windows version.

Regards,

Jo

Jo,

It's possible it could be a low memory issue within Terragen 2 itself.  But, the operating system as a whole has 4G installed, only 1.6G of which was in use at the time TG2 was running (1.5G of it being TG2).  So, if it had been a global system memory issue (which I'm not sure if that's what you mean), then I would have had troubles with other applications.   However, that wasn't the case.  At the time TG2 was running, I copied the screen shots to the clipboard, launched the Gimp to create the attached jpeg images, pasted them from the clipboard into the Gimp, saved the files and exited all without any GUI troubles.  In other words, there was plenty of system memory available for launching other tasks on the system above what TG2 was doing.   

Also, while the VM memory was approaching the 2GB limit (2,056,268K) , it hadn't actually reached that point (unless the limit is slightly less than (1024^2)*2).  The Working Set (which is the actual memory in use by the app) was 1.5GB at 1,524,536K (still .5 GB under the 2GB ceiling).  Also, while the handles may have been under what you would have expected at 373, I'm still not discounting GDI issues.  In the past, when I've seen apps lose buttons, menus and other GUI interface controls, I've found this related to GDI troubles when the system still shows ample ram.  But, a low memory situation could cause both the GDI issues and the threads to crash.

Of course, I don't know how Windows counts memory towards to 2GB limit.  If it's only what's in the working set, then it was well under that limit.  However, if it tallies in vm + working set, then total memory use would have been well over 2GB.

After reading more about GDI, I found that GDI allocates its objects from the kernel memory pool which comes from paged pool memory.  So, I would have to assume that the paged pool memory is running out and GDI can't create new objects.  Considering that the VM maximum was so close to the 2GB limit (and possibly even fragmented), there may not have been enough memory space left to create new GDI objects.  It does appear, though, that Vista is isolating each app from one another.  So, while one app may run out of internal space, it doesn't appear to affect the rest of the OS or other apps.

As soon as I get home, I will upload my project, but it will take me a little bit of time to gather up all of the pieces (models, textures and whatnot).

Thanks.

commorancy

Quote from: jo on October 03, 2008, 10:39:10 PM
Hi Brian,

Can you please pack up all the files used in your scene and email them to me, or if it's getting close to 10 MB could you please upload them somewhere and send me a link? I'd like to try your scene with the Mac version and the Windows version.

Regards,

Jo

I have sent you an email containing a link to download the project Zip file.  Please let me know if you didn't receive this email.

Thanks.

--
Brian

rcallicotte

I'm having the same issue.  What I have figured out is if I render a small render, everything is fine.  If I make a larger render, then I get these errors.  Same settings, but higher quality and larger means getting this error.


  • Better Settings
  • Larger Render
So this is Disney World.  Can we live here?

freelancah

I changed from 32bit WIN XP to 64bit Vista and so far I've been able to render 1  of my previous works at full hd resolution without any errors. I will continue testing to see if I was just lucky or that the problem actually disapeared..

commorancy

Quote from: freelancah on October 06, 2008, 11:28:44 PM
I changed from 32bit WIN XP to 64bit Vista and so far I've been able to render 1  of my previous works at full hd resolution without any errors. I will continue testing to see if I was just lucky or that the problem actually disapeared..

Hmm... so Terragen 2 runs on 64 bit Vista?  It does look like Vista 64 will run most 32 bit apps without issues, but I'm not sure it expands the internal memory space of a 32 bit app.  I'll have to upgrade my system to 64 bit and give it a try.

Thanks.

rcallicotte

This is sort of off-task, but what about all other applications?  Do these run on Vista 64 without trouble?
So this is Disney World.  Can we live here?

commorancy

#29
Quote from: calico on October 07, 2008, 09:51:18 AM
This is sort of off-task, but what about all other applications?  Do these run on Vista 64 without trouble?


I would think it depends on the app.  I'll have to give it a try and see how far it goes.  Since the OS takes care of launching and running an app, there is a layer there that can let a 32 bit app run.   Drivers are the most problematic portion as they have to be designed for 64 bit.  I may end up downgrading back to 32 bit on this system if too many things don't end up working.  However, I read elsewhere that Vista 64 runs most 32 bit apps without issue even if not specifically designed to run on 64 bit Vista.  So, I'm hoping most of my apps will work.  I guess I'll find out once I upgrade.  I know that Poser will work, but I'm not sure about the rest.

However, on my HP system, it does appear that HP fully supports 64 bit Vista for my 6600 quad core system.  So, all of the hardware should have fully working drivers for 64 bit Vista.  So, my concern is not so much the drivers, but the apps.

Thanks.

--
Brian