Hello,
I'm seeing a problem where, during a render, one or more of the threads crash with 'An unknown error occurred in a render thread'. In my first instance of this, one of four render threads died (4 cpu system). When the thread died, this left the other 3 threads running. The problem is that when the thread dies, it leaves the tile it was working on unfinished. No other thread realizes this error to go back and finish that tile. Instead, the tile where the thread died remains unrendered (black or mostly black). Since I've found no way to tell Terragen 2 which tile to work on specifically, there's no way to, other than guessing by using the Crop function, to get that area rendered.
The second time I've seen this issue, all four render threads failed with that same error as above. So, it appears that when a thread dies, it does not restart itself. Instead, the work it was doing is left unfinished and the thread does not restart. This issue leaves the render window open hanging waiting for the render to complete, yet it will never complete because the threads died. In the case of all 4 threads dying, I had to manually kill the render because the render window apparently doesn't keep track of thread failures.
I would suggest that if a thread dies, that either 1) stop the entire render process or 2) restart the thread picking up where the dead thread left off. It would be best to attempt to restart the thread and let it continue to render. However, if the thread continues to fail, then close the entire render. The way it works now, it's impossible to complete the render of a scene when the threads randomly die. I would also suggest hooking up the render window to the status of the threads. If all threads fail, the render window should mark the status of the render as finished with errors.
This issue doesn't happen until the second stage of the render begins. So, it gets all the way through the lengthy dotty tile sequence and then these errors occur during the second stage of rendering (when it begins to fill in the tiles with the final image).
Note that this is in TP5 (1.9.99.1) operating on Windows Vista with a quad core and 3G of ram.
Thanks.
--
Brian
Additionally, I should point out that this scene is also generating population errors described in another thread (http://forums.planetside.co.uk/index.php?topic=4320.0). I do not know if the population errors are related to the thread crashing.
Thanks.
--
Brian
Wouldn't be better if instead of shutting down the whole render if a thread fails as you suggest (Thank you...but no) if Terragen was smart enough to use a technique similar to some high end render farm software that is constantly looking at the state of each machine on the farm and should one stop responding or no longer be on the farm for what ever reason, that machines current work state is moved to a machine on the farm not currently in use.
One would think that in theory Terragen would look at the states of the threads it is using, and should a thread fail for what ever reason shunt that threads workload gracefully as an automatic process as part of its core to another thread: this would be the preferred solution to termination of the render with out explanation at least form a "End User Frustration" point of view as if you've had a render going for some hours, and then it terminates with out explanation or even force the application to crash to desktop there would be a great many enquirers, as to why here on these forums.
Regards to you.
Cyber-Angel
Quote from: Cyber-Angel on August 01, 2008, 07:56:40 PM
Wouldn't be better if instead of shutting down the whole render if a thread fails as you suggest (Thank you...but no) if Terragen was smart enough to use a technique similar to some high end render farm software that is constantly looking at the state of each machine on the farm and should one stop responding or no longer be on the farm for what ever reason, that machines current work state is moved to a machine on the farm not currently in use.
Yes, I would also prefer the renderer to start up a new thread. The thread that died would leave a new processor available to start new work. So, in fact, I agree that it should start a new thread to continue the unfinished work and move on. That said... if there is an unresolvable issue in the data set supplied to the renderer, then there is no other choice than to abort the entire render after that set of conditions have been met. That means implementation of an iteration system to count thread failures over a given bit of data. If the thread continues to fail and restart over and over, that's just as bad as dying and not restarting. Thus, if the data cannot be rendered after a certain amount of tries, the renderer must give up and notify the user.
We all know that there are scenarios that can be created through the workflow functions that can lead to loops and other unrenderable data sets. So, this is why it's important for them to create a retry system and eventually give up the render if the threads die unexpectedly due to the data set.
It's actually more important to know why a thread failed than to restart the render after a failure. If the thread can give some kind of information on why it failed in the errors area that's user friendly, the user can then adjust the render data to allow it to render.
Quote
One would think that in theory Terragen would look at the states of the threads it is using, and should a thread fail for what ever reason shunt that threads workload gracefully as an automatic process as part of its core to another thread: this would be the preferred solution to termination of the render with out explanation at least form a "End User Frustration" point of view as if you've had a render going for some hours, and then it terminates with out explanation or even force the application to crash to desktop there would be a great many enquirers, as to why here on these forums.
Cyber-Angel
To be perfectly honest, what you are asking for here is a preparsing system. A preparsing system could go through the render input data and do integrity checking, before beginning the actual render, to warn of issues. A parsing system could help the user clean up the scene and prevent failures hours into a render.
Thanks.
--
Brian
You could get round the loop problem by having "Loop Cheeking" as part of the core but since I'm not a programmer and know vary little about this subject apart form some extremely elementary knowledge: that is extremely vague and hazy in terms of my recollection of it, I could no say where "Loop Cheeking" would be used other than maybe have it look for loops at the parsing stage, if such technologies where to be implemented in the future.
Regards to you.
Cyber-Angel
I've found a reasonably effective workaround for both the population errors and the thread crashing issue. I set the crop to a reasonably small horizontal stripe. I render each stripe going from top to bottom of the image. So far it looks like about 7-10 stripes for the entire image at 1920x1080. This process takes longer only because it's a manual process, but each stripe render is reasonably short, creates an incremental render (so you don't lose everything in case of error) and seems to get around whatever the issue that causes the thread crashing and population errors when trying to render the whole screen all at once.
So, I would like to see something like this added to Terragen. Instead of trying to render the entire image size all at once, break the image down into a bunch of crop sections and render each crop section separately (stage 1, then stage 2 for only that section) and incrementally save that work. Then move to the next section and begin the next section. When attempting a full sized render, I'm concerned that this issue is related to some Windows resource issue (ram, thread constraints, etc) when combined with the memory needed for a very large scene. This stripe rendering methodology seems to get around whatever resource limitation is causing these thread failures and population errors. Incremental saves could allow for restarting a rendering at a previously saved point if it aborts from error.
Of course, at the end it all has to be stitched together in a final image, but that's reasonably easy with something like the Gimp or Photoshop. This stitching process could also be automated.
Thanks.
--
Brian
A note about rendering separate crops. While I was rendering my image in small crops to avoid the thread crashing and population errors, I found another issue. The rendered crops do not retain the exact color balance as previous rendered crops. So, when you stitch the image back together, some of the crops end up a slightly different color balance than other rendered crops. This leaves noticeable stitch 'lines' between the various crops when joined into the final image. I found that I had to color correct some of the crops to get them to match other crops. The odd thing is, it didn't seem to affect all parts of the image. It affected, primarily, the haze color value. The foreground colors seemed not to be altered that I could tell.
Just an FYI in case you want to try the crop + stitching process to avoid population errors and thread crashing.
--
Brian
Quote from: commorancy on August 03, 2008, 10:46:05 AM
A note about rendering separate crops. While I was rendering my image in small crops to avoid the thread crashing and population errors, I found another issue. The rendered crops do not retain the exact color balance as previous rendered crops. So, when you stitch the image back together, some of the crops end up a slightly different color balance than other rendered crops. This leaves noticeable stitch 'lines' between the various crops when joined into the final image. I found that I had to color correct some of the crops to get them to match other crops. The odd thing is, it didn't seem to affect all parts of the image. It affected, primarily, the haze color value. The foreground colors seemed not to be altered that I could tell.
Just an FYI in case you want to try the crop + stitching process to avoid population errors and thread crashing.
--
Brian
This is an old and well known problem since the very beginning of TG2 and is due to the GI. Though it is improved quite much. I only notice seams when rendering at high quality settings with minimum GI of 2/2 and then it is rather mostly an exception.
I'm going to have Matt come and take a look at these issues, but here's currently attending Siggraph so it may be a few days before he will be able to get to it.
- Oshyan
Quote from: Oshyan on August 12, 2008, 03:33:53 AM
I'm going to have Matt come and take a look at these issues, but here's currently attending Siggraph so it may be a few days before he will be able to get to it.
- Oshyan
Wow, Matt's a lucky bastard ;D
I suppose he hasn't an oral there about TG? At least, I didn't find it quickly at CGSociety.
Martin
No presentations this year I'm afraid. ;D
- Oshyan
Any news on this subject yet? I'm having the same problem rendering at 1920x1200. Always crashes at the end but if I render in 4 different cropped pieces it works fine only problem is that the colours are not exactly the same on the cropped pieces so I have to do some editing to nit the pieces up
Quote from: freelancah on September 28, 2008, 07:58:10 AM
Any news on this subject yet? I'm having the same problem rendering at 1920x1200. Always crashes at the end but if I render in 4 different cropped pieces it works fine only problem is that the colours are not exactly the same on the cropped pieces so I have to do some editing to nit the pieces up
I'm still having this error myself. In fact, with a new scene I've found these additional issues (usually 20+ times):
An unknown error occurred in TraceRay
An unknown error occurred in trBucketRender: RenderMore() ray-traced pass
It also appears that the thread with this error dies with:
An unknown error occurred in render thread (one for each thread that dies).Then, the only way to resolve this issue is to close TG2 and reopen it. If you don't restart TG2, the errors persist on subsequent renders. Restarting TG2 appears to be the only way to clear out the state of the renderer (or whatever variable states are still out there). This is a hassle because I have to regenerate heightfields manually and regenerate populations manually before I can render again.
Thanks.
Here's a screenshot what's happening when the rendering fails. http://www.joce-servers.com/~freelancah/Error.jpg
I'm using C2D Quad 9400 with 4gb of memory. The same error occurs on my old C2D E6750 so I'm pretty sure it's not the system setup
Quote from: freelancah on October 02, 2008, 06:00:38 AM
Here's a screenshot what's happening when the rendering fails. http://www.joce-servers.com/~freelancah/Error.jpg
I'm using C2D Quad 9400 with 4gb of memory. The same error occurs on my old C2D E6750 so I'm pretty sure it's not the system setup
It also happens on my Quad 6600 also with 4GB of RAM. However, in a previous forum thread it was stated that because TG2 is a 32 bit app, it can only address a max of 2GB of memory anyway. My thought is that this may be a memory related issue. It could simply be running out of memory while rendering. If so, then I'd like to see a 64 bit version of TG2. I'd be willing to install 64 bit Windows to get better performance and more application addressable RAM. But, only someone from Planetside can explain if this is the issue and whether a 64 bit version of TG2 would resolve this issue.
BTW, those billowy clouds look amazing in the pre-render. I'd like to see your render fully completed.
Thanks.
Yeah I guess it could be that it runs out of memory. This doesn't happen if I render on lower resolutions, like 1280x768 and I can also bypass the problem on highres by cropping but still I feel this is a big problem to not being able to render one whole image.
BTW I also tried with just 1 thread rendering and the same problem occured so atleast it's not related to the number of threads
Quote from: freelancah on October 02, 2008, 02:16:15 PM
BTW I also tried with just 1 thread rendering and the same problem occured so atleast it's not related to the number of threads
It doesn't seem to be related to the number of threads. It could, however, be related to the multithreading support that was added. I'm interested to find out if the scene renders in the previous version that didn't have multithreading enabled. I seem to remember rendering full sized complex 3D scenes on the non-multithreaded version without these issues (obviously, very slowly). I'm betting these renders would work on the non-multithreaded version. So, I'm leaning towards a resource issue related to memory and/or multithreading (possibly a combination).
I'm thinking of rolling back to that version and test rendering with that version. I'd much rather have it render completely in one pass than have to piece it all together later using tiny crops.. even if it takes several days to render.
Thanks.
--
Brian
Quote from: freelancah on October 02, 2008, 02:16:15 PM
BTW I also tried with just 1 thread rendering and the same problem occured so atleast it's not related to the number of threads
Update: I tried rolling back to the newest non-threaded version, but that version doesn't support water subsurface transparencies (which I'm using). There's not a shader with transparencies in that version. So, I'm stuck using the threaded versions for this render. You might be able to use it on your render if you're not using transparencies.
Thanks.
--
Brian
Brian, try setting max threads to 0. I believe this still roughly emulates the single-threaded rendering of the past.
It does seem like this may be a memory issue. The best way to tell is to monitor memory use for TG2 up until the moment it crashes. You can use Task Manager for this if you're willing to watch it (or perhaps check after it has crashed as it probably hasn't released the memory yet), or you can use more sophisticated long-term logging tools (one built-in to Windows is Perfmon, the Windows Performance Monitor, accessed through the Administrative Tools).
- Oshyan
Quote from: Oshyan on October 02, 2008, 11:41:42 PM
Brian, try setting max threads to 0. I believe this still roughly emulates the single-threaded rendering of the past.
It does seem like this may be a memory issue. The best way to tell is to monitor memory use for TG2 up until the moment it crashes. You can use Task Manager for this if you're willing to watch it (or perhaps check after it has crashed as it probably hasn't released the memory yet), or you can use more sophisticated long-term logging tools (one built-in to Windows is Perfmon, the Windows Performance Monitor, accessed through the Administrative Tools).
- Oshyan
Thanks for the tip. I will try a render with max threads set to 0 and let you know. I sometimes see the error in the pre-render stage, but sometimes it won't happen until the final render stage begins. I'll let you know how it goes.
I will say that Poser must have had this issue as well and resolved it. What it looks like they did was allow the user to set the max 'bucket' size (thread render tile size). Apparently, the smaller the bucket size, the smaller the memory footprint to render each tile. It might be worth a try to allow the user decrease the tile size for rendering. This may then require less memory to render each tile and allow larger sized images to complete. Right now it looks like TG2 arbitrarily picks the tile size based on some screen dimension calculation combined with GI settings.
I'm running a render with Process Explorer watching. So far, the working memory set is up to 1,368,356K and growing. The virtual size is 2,058,296K. It hasn't stopped yet. Although, I will admit I reduced some GI parameters and the Detail setting in this render. If/when it crashes, I will take a snapshot of the Process Explorer memory window and post it here.
Update: First try, it got about 3/4 through stage 1 pre-render and then I got the C++ error and the whole thing closed. This usually happens when I don't preallocate the subdiv cache. But, I had that set at 400. So, not sure about why I got the C++ error. The memory wasn't over 1.4MB working set when it crashed. So, that didn't appear to be memory related (at least, running out). I'm going to try the render again at my original settings.
Thanks.
Oshyan,
After running an interesting test with Process Explorer, I'm starting to believe that the threads dying may be a thread synchronization issue. While running Process Explorer and monitoring the memory stats for tgd.exe, the error did not occur and the render had gotten farther than it had ever gotten without crops. I know that monitoring a process slows the process down during each data gathering iteration. The performance monitoring obviously slows down the render threads as well, but the threads did not crash as long as Process Explorer was monitoring the process. Out of curiosity, I quit Process Explorer during the monitor of the render (I wanted to see if it would speed up the render). Within 5 seconds of closing Process Explorer, I had a message that a thread had died.
Note that when the thread died, the memory was still well below 2GB (around 1.2-1.3GB in the Working Set).
I'm restarting the render again and I'm planning on leaving Process Explorer running the entire time (or until an error occurs with PE running). However, the fact that PE is slowing the process and render threads down without errors indicates something other than a memory issue. My guess is something to do with thread synchronization.
Update: Checking this morning, 3 threads have died, One is still working. See the attached Memory and Messages panels. I clicked 'Stop' in the rendering window and the smaller confirmation window tries to appear. See EmptyPanel.jpg for example. The contents of the cancel panel window do not draw. While it appears that the window content is not there, in fact it is.. the content was just not drawn. I was able to randomly click and find the button to get out of the panel. However, note that when Window contents do not draw (like buttons and menus), this usually indicates a GDI resource issue. I do not know if the GDI resource issue caused the threads to fail, but it's very possible since you are displaying bitmaps to the screen via the threads.
Thanks.
--
Brian
Hi Brian,
It looks to me like a bit like you're running out of memory. I don't think the problem is a GDI leak because from the Process Explorer it seems like the number of GDI handles being used is about what I would expect. The reason the dialog is blank could also be down to running out of memory.
Can you please pack up all the files used in your scene and email them to me at jomeder@ihug.co.nz, or if it's getting close to 10 MB could you please upload them somewhere and send me a link? I'd like to try your scene with the Mac version and the Windows version.
Regards,
Jo
Quote from: jo on October 03, 2008, 10:39:10 PM
Hi Brian,
It looks to me like a bit like you're running out of memory. I don't think the problem is a GDI leak because from the Process Explorer it seems like the number of GDI handles being used is about what I would expect.
Can you please pack up all the files used in your scene and email them to me, or if it's getting close to 10 MB could you please upload them somewhere and send me a link? I'd like to try your scene with the Mac version and the Windows version.
Regards,
Jo
Jo,
It's possible it could be a low memory issue within Terragen 2 itself. But, the operating system as a whole has 4G installed, only 1.6G of which was in use at the time TG2 was running (1.5G of it being TG2). So, if it had been a global system memory issue (which I'm not sure if that's what you mean), then I would have had troubles with other applications. However, that wasn't the case. At the time TG2 was running, I copied the screen shots to the clipboard, launched the Gimp to create the attached jpeg images, pasted them from the clipboard into the Gimp, saved the files and exited all without any GUI troubles. In other words, there was plenty of system memory available for launching other tasks on the system above what TG2 was doing.
Also, while the VM memory was approaching the 2GB limit (2,056,268K) , it hadn't actually reached that point (unless the limit is slightly less than (1024^2)*2). The Working Set (which is the actual memory in use by the app) was 1.5GB at 1,524,536K (still .5 GB under the 2GB ceiling). Also, while the handles may have been under what you would have expected at 373, I'm still not discounting GDI issues. In the past, when I've seen apps lose buttons, menus and other GUI interface controls, I've found this related to GDI troubles when the system still shows ample ram. But, a low memory situation could cause both the GDI issues and the threads to crash.
Of course, I don't know how Windows counts memory towards to 2GB limit. If it's only what's in the working set, then it was well under that limit. However, if it tallies in vm + working set, then total memory use would have been well over 2GB.
After reading more about GDI, I found that GDI allocates its objects from the kernel memory pool which comes from paged pool memory. So, I would have to assume that the paged pool memory is running out and GDI can't create new objects. Considering that the VM maximum was so close to the 2GB limit (and possibly even fragmented), there may not have been enough memory space left to create new GDI objects. It does appear, though, that Vista is isolating each app from one another. So, while one app may run out of internal space, it doesn't appear to affect the rest of the OS or other apps.
As soon as I get home, I will upload my project, but it will take me a little bit of time to gather up all of the pieces (models, textures and whatnot).
Thanks.
Quote from: jo on October 03, 2008, 10:39:10 PM
Hi Brian,
Can you please pack up all the files used in your scene and email them to me, or if it's getting close to 10 MB could you please upload them somewhere and send me a link? I'd like to try your scene with the Mac version and the Windows version.
Regards,
Jo
I have sent you an email containing a link to download the project Zip file. Please let me know if you didn't receive this email.
Thanks.
--
Brian
I'm having the same issue. What I have figured out is if I render a small render, everything is fine. If I make a larger render, then I get these errors. Same settings, but higher quality and larger means getting this error.
- Better Settings
- Larger Render
I changed from 32bit WIN XP to 64bit Vista and so far I've been able to render 1 of my previous works at full hd resolution without any errors. I will continue testing to see if I was just lucky or that the problem actually disapeared..
Quote from: freelancah on October 06, 2008, 11:28:44 PM
I changed from 32bit WIN XP to 64bit Vista and so far I've been able to render 1 of my previous works at full hd resolution without any errors. I will continue testing to see if I was just lucky or that the problem actually disapeared..
Hmm... so Terragen 2 runs on 64 bit Vista? It does look like Vista 64 will run most 32 bit apps without issues, but I'm not sure it expands the internal memory space of a 32 bit app. I'll have to upgrade my system to 64 bit and give it a try.
Thanks.
This is sort of off-task, but what about all other applications? Do these run on Vista 64 without trouble?
Quote from: calico on October 07, 2008, 09:51:18 AM
This is sort of off-task, but what about all other applications? Do these run on Vista 64 without trouble?
I would think it depends on the app. I'll have to give it a try and see how far it goes. Since the OS takes care of launching and running an app, there is a layer there that can let a 32 bit app run. Drivers are the most problematic portion as they have to be designed for 64 bit. I may end up downgrading back to 32 bit on this system if too many things don't end up working. However, I read elsewhere that Vista 64 runs most 32 bit apps without issue even if not specifically designed to run on 64 bit Vista. So, I'm hoping most of my apps will work. I guess I'll find out once I upgrade. I know that Poser will work, but I'm not sure about the rest.
However, on my HP system, it does appear that HP fully supports 64 bit Vista for my 6600 quad core system. So, all of the hardware should have fully working drivers for 64 bit Vista. So, my concern is not so much the drivers, but the apps.
Thanks.
--
Brian
Thanks commorancy. If you wouldn't posting here or even in the Open Discussion session what your experience is, I'd really like to soak up that data. I play games and use Poser and other 3D modeling programs and even DAZ Studio.
Quote from: calico on October 07, 2008, 12:27:25 PM
Thanks commorancy. If you wouldn't posting here or even in the Open Discussion session what your experience is, I'd really like to soak up that data. I play games and use Poser and other 3D modeling programs and even DAZ Studio.
I've ordered everything I'll need to upgrade my system and hopefully it will all be here Thursday. I plan to spend this coming weekend on the upgrade project. It's too bad that it requires a full reinstall of everything (including apps). There's no easy upgrade method from 32 bit to 64 bit. So, that's why I expect it to take the entire weekend to get everything reinstalled and all my projects restored.
Thanks.
--
Brian
Both XP x64 and Vista x64 support running the vast majority of 32 bit apps without any problem. In addition, unlike 32 bit versions of Windows, 64 bit versions can allow a 32 bit application to take a full 4GB of memory to itself, provided that application has been compiled to be "large address aware" (which recent builds of TG2 have been). So I think if memory issues are holding you back, you will find TG2 under x64 a lot more capable. I use it on Vista x64 with 8GB of RAM myself and I've been able to do some insanely detailed scenes since the "large address support" was enabled - 10's of millions of detailed tree instances, *trillions* of polygons...
- Oshyan
Let's see some of these, if you can.
Quote from: Oshyan on October 07, 2008, 10:37:35 PM
...I've been able to do some insanely detailed scenes...
- Oshyan
I haven't had time to render another scene, but as Oshyan said: in Vista terragen can use up to 4gb of memory. In the first scene I was able to render the memory usage was around 2.85gb so I guess that was the problem atleast on my part
Quote from: calico on October 08, 2008, 08:33:43 AM
Let's see some of these, if you can.
Quote from: Oshyan on October 07, 2008, 10:37:35 PM
...I've been able to do some insanely detailed scenes...
- Oshyan
They aren't particularly exciting, just tests of high object counts. I'll show when I have something interesting. ;)
- Oshyan
Note, setting Max Threads to 0 and then rendering resolves the memory / thread crashing issue. Clearly, TG2 renders much slower, but the single thread does not error out. I ran a recent render where one thread crashed when using 4 threads. Yet, when Max Threads value is set to 0, it renders without error. One thing I did to try and speed up the render a little was force the affinity to one of the CPUs, then set the priority to 'Realtime'. This way, it maxes out that one CPU of the four.
I haven't tried limiting the threads to 2 or 3 to see if that reduces the chance of error.
Thanks.
--
Brian
Balloon Lake
(http://farm4.static.flickr.com/3161/2926110907_9944334c90.jpg?v=0)
Flickr Page: http://www.flickr.com/photos/commorancy/ (http://www.flickr.com/photos/commorancy/)
Please do try limiting the threads to 1 or 2. It may still be a memory issue since using more cores will take up more memory.
- Oshyan
I'll give the two thread render a try next time I'm in 32 bit Vista. I got the upgrade kit on Thursday and decided to upgrade it. It actually went a lot faster than I thought. Anyway, I'm running one if the renders on 64 bit Vista that was crashing on 32 bit Vista to see if it will now render. As soon as it gets done, I'll boot back into 32 bit and try a 2 thread render.
Thanks.
--
Brian
As an update... here's the status of the 64 bit Vista install.
I was able to get Vista 64 installed. I re-rendered one of the previous scenes where threads kept dying in 32 bit Vista. In 64 bit Vista, the render succeeded without any issues using all 4 threads. Note that I installed not only Vista 64, but an additional 4GB of RAM for a total of 8GB of memory in the system. So, apparently, with the scenes I was creating, somewhere along the way during the render, Terragen 2 is running up against a 32 bit memory limitation. I'd like to see a fully native 64 bit compiled Terragen 2 to take full advantage of the 64 bit system.
However, there is an OpenGL issue under 64 bit Vista that isn't present, to this degree, in 32 bit Vista. The OpenGL library of the 8400GS Nvidia card drivers (in my system) is VERY buggy in 64 bit Vista. Some OpenGL bugs also existed in 32 bit Vista, but did not affect Terragen 2. However, in 64 bit Vista, simply moving around in the node network (dragging the screen around) is enough to crash the Nvidia OpenGL library and, by association, Terragen 2. This crash also leaves the entire video driver in an unstable state which requires a reboot. This OpenGL crashing issue is a known issue that Nvidia has been working on for several months now (and has been unable to resolve). I ran into this issue with Poser originally. Poser, however, thankfully supports SreeD (a software based OpenGL-type engine). So, I can use Poser without crashing.
Note that I have the most recent Nvidia drivers installed from 9/25/2008 and these bugs are still present.
Apparently, this OpenGL driver crashing issue may affect many cards in the 8XXX series of Nvidia GPUs. It may also affect other Nvidia card models outside of the 8XXX series. The extent of this OpenGL bug in the Nvidia drivers is presently unknown. According to SmithMicro (Poser), Nvidia is aware of the problem and they are working to resolve it.
At this point, I'm considering the purchase of another GPU for my system... most probably an ATI card where this OpenGL issue is likely not present. I'll just need to find one that has HDMI and supports 64 bit Vista.
Thanks.
--
Brian
Good to hear you resolved your rendering problem too. I just finished a second render succeffully at 1920x1200 resolution, so working fine here too.
I have a 8800 GT 512mb and I dont have the opengl bug you are describing and I really dont know if this is any help but you might wanna try these: http://www.omegadrivers.net/ they are slightly customised drivers which with luck, might work.
Quote from: freelancah on October 11, 2008, 06:12:53 PM
Good to hear you resolved your rendering problem too. I just finished a second render succeffully at 1920x1200 resolution, so working fine here too.
I have a 8800 GT 512mb and I dont have the opengl bug you are describing and I really dont know if this is any help but you might wanna try these: http://www.omegadrivers.net/ they are slightly customised drivers which with luck, might work.
I'll take a look at those drivers. However, searching '8800GT Crash OpenGL' on Google leads me to a post on NVidia's forum from a user complaining of crashes from nvoglnt.dll on Win XP when using Quake. This post was from back in Feb 2008 and driver updates have been released since then. But, this also indicates that the 8800GT may be susceptible to at least some level of OpenGL bugs. Perhaps not the one that crashes Terragen, but it doesn't appear immune. I've also found the OpenGL drivers to be a bit more stable under a 32 bit OS than a 64 bit OS. Is your 8800GT installed on your 64 bit Vista system?
I've also seen posts from users complaining of OpenGL crashes under MacOS X and Linux when using cards in the 8000 series. Again, these drivers may have been updated since then. Even still, my 8400GS still has these bugs even with the latest driver. So, you'd think that if there are crashing problems being submitted to forums around the net related to OpenGL that Nvidia engineers would be working to fix it across the board. I'm not seeing this from Nvidia. A simple Google search reveals people experiencing OpenGL library crashes under Windows using 8000 series cards.
Thanks.
--
Brian
Quote from: freelancah on October 11, 2008, 06:12:53 PM
I really dont know if this is any help but you might wanna try these: http://www.omegadrivers.net/ they are slightly customised drivers which with luck, might work.
I have the Omega drivers installed now. Poser still crashes in the same way when OpenGL preview is enabled, but now it says nvlddmkm is at fault and not nvogl. Terragen 2 also triggers this crash and then closes as well. BTW, this crash has the exact same appearance as the Nvidia supplied driver's crash, it's just that a different component is said to be crashing. This bug is triggered in the exact same way and under the exact same circumstances as the OpenGL problem. It appears that the Omega drivers are able to recover far better from this situation and not leave the drivers in an unstable state, though.
I am also seeing users posting about nvlddmkm crashing on Nvidia's forum (http://forums.nvidia.com/index.php?showtopic=25381 (http://forums.nvidia.com/index.php?showtopic=25381)) in 2007 without any real solution. This issue appears to affect cards in at least 6000, 7000 and 8000 series and appears to only affect Vista. It also appears that Nvidia may not have fixed this original problem and, instead, pushed the problem off to a different component (OpenGL) so that it appears to be an entirely new bug.
Ok, after having dug some more into this problem, it may be a hardware and/or memory related issue combined with Vista. See http://www.ocztechnologyforum.com/forum/showthread.php?t=32703 (http://www.ocztechnologyforum.com/forum/showthread.php?t=32703). It appears that this may have something to do with memory and chipset timings. As part of this upgrade, I also installed DDR2 800 speed memory. I upgraded it from 667 speed. Although, I was still having OpenGL crashing issues even with 667 memory and even under 32 bit Vista. Apparently, XP may not suffer from this exact problem.
I'm going to keep poking around on this one... although, maybe I'll get an ATI just for kicks (to see if it fixes the issue).
--
Brian
So you have both 667 and 800 mhz memory running at 800mhz? With the timings of the old memory (that is if you didn't remove them before installing new ones). If so you should try to loosen the timings and/or lower the frequency to 667 mhz. Memory running at lower frequencys tend to have tighter timings than memory running at high freq. So that might be the issue if the new memory isn't able to handle the timings. Why it happens only on vista is something I dont know but it's worth checking.
Sorry for the messy text. I should be sleeping but still testing on few renders :D
Quote from: freelancah on October 11, 2008, 10:50:39 PM
So you have both 667 and 800 mhz memory running at 800mhz? With the timings of the old memory (that is if you didn't remove them before installing new ones). If so you should try to loosen the timings and/or lower the frequency to 667 mhz. Memory running at lower frequencys tend to have tighter timings than memory running at high freq. So that might be the issue if the new memory isn't able to handle the timings. Why it happens only on vista is something I dont know but it's worth checking.
Sorry for the messy text. I should be sleeping but still testing on few renders :D
I took out the 667 and replaced it with 800. So, it's all 800. The 4GB I previously had in there was all 667 (1GB sticks). There's only 4 slots anyway, so I had to replace it all to get 8GB in there (2GB sticks). I haven't tried going back to the 667 and running the crashing tests with only 4GB of memory. I'm not sure it will affect anything, though. I was still having OpenGL crashing issues even on the 4GB of 667 on Vista 32. Can't remember trying it on Vista 64 before upgrading the RAM. So, I'm not entirely sure it's a RAM + Timing issue. It's not as if HP gives me fine grained control over the MB timings anyway. So, basically, the MB just does whatever it does.
If it is the 800 that is the problem, the only solution is to replace it with 2GB sticks at 667. But, I'm still not convinced that it's RAM timing issue. I ran Memtest+ and it fully passed (took almost 2 hours to complete). I need to contact HP and make sure I have the absolute latest MB chipset drivers on this system. I think I do, but HP's site is just not always the best at keeping up-to-date.
Thanks.
As a test, I replaced the 8400GS card with an ATI Radeon 3650 w/ 1GB GDDR2. I had to download the drivers from AMD directly as Diamond (card manufacturer) didn't have the Vista 64 drivers on their site. I installed the latest Catalyst suite that I downloaded and the card is working perfectly (for the moment). Both Poser and Terragen 2 have no issues with OpenGL any longer. The card supports HDMI and comes with a 5.1 audio driver, so that works for me.
I didn't buy the most expensive card as it would have required a 500 watt power supply and I didn't want to go through that hassle tonight. This one works with my existing 300W supply. So, for the moment, I'm quite happy with the purchase. I might upgrade to a more expensive card later, but I won't be doing much gaming with this card and it should work well enough for the games I want to play.
Ok, so I spoke too soon. While Poser seems to work with OpenGL (for the most part), I am still having issues with Terragen 2. If I move the screen around the node area, the display driver stops responding and recovers in a very similar way to the Nvidia 8400GS card. This leads me to believe this may be a 64 bit Vista issue. It may be related to SP1. Alternatively, it may be some kind of strange bug between the 32 bit subsystem and the 64 bit graphics drivers. Since both ATI and Nvidia cards behave in an almost identical way with completely different drivers and hardware, I'm leaning towards a Vista operating system problem as the root cause. I find it unlikely that two completely different hardwares with two completely different drivers to have identical bugs. The commonality, then, is the OS and the motherboard / hardware. So, barring a fundamental design flaw in the motherboard (not likely) or a problem with the RAM (also not likely), the only thing left then is the OS.
I have found an MS KB article (http://support.microsoft.com/kb/938194) that describes a fix for 'video display driver not responding issue'. I tried to apply it and the update told me it doesn't apply to my system. So, I assume this fix is already in SP1. I've also seen fixes not completely resolve the root problem. So, I believe this is a Vista issue that is still unresolved. Maybe MS will fix it in SP2.
--
Brian
The good news, though, is that the 64 bit environment does resolve the 32 bit memory and thread crashing issues I was having. The bad news is that Vista 64 is unstable when it comes to video drivers (no surprise there). I'm happy that I can render larger images, but the crashing video driver thing is quite annoying.
I'm wondering if a natively compiled 64 bit version of Terragen 2 would experience these video driver issues? My guess is that it may not. So, is there any possibility of a native 64 bit version of Terragen 2?
Thanks.
Have you installed all the latest vista upgrades? You also might wanna try updating your bios and every 64bit driver you can find. I can't really think of anything else than drivers or bios that can be causing this, since you already changed few components
Hi Brian,
We're going to look at doing 64 bit versions after the TG2 final release.
Regards,
Jo
Quote from: freelancah on October 12, 2008, 12:26:10 PM
Have you installed all the latest vista upgrades? You also might wanna try updating your bios and every 64bit driver you can find. I can't really think of anything else than drivers or bios that can be causing this, since you already changed few components
I've updated everything I can get my hands on including the chipset drivers, video drivers, bios, windows, etc. There's nothing left to update. The reason I believe this to be a Windows issue is that I found a previous Microsoft KB article that's supposed to address this exact issue 'video driver not responding' (KB938194 (http://support.microsoft.com/kb/938194)). Apparently, the hotfix for this issue has been rolled into SP1 now. But, it appears this update may not have completely fixed this issue. Considering that this issue is present on two different video cards with two different video drivers, I have to rule out the drivers as the fundamental cause. My thought is that this is a Microsoft issue in Vista (or possibly in just Vista 64). It may even be related to the 32 bit application connectors to the 64 bit drivers.
The only other issue is that this is simply bad hardware, but I have no reason to believe that this is the case as it only affects a very specific application situation. If it were bad hardware, I would expect to see a lot more random problems across the board than a very specific reproduceable trigger.
Quote from: jo on October 12, 2008, 04:58:39 PM
We're going to look at doing 64 bit versions after the TG2 final release.
Jo
Thanks Jo. I'll be looking forward to it.
-Brian
I run Vista x64 with an 8800 GTS with no problems like this. I don't think it is a fundamental TG2 flaw, nor necessarily even a driver flaw since I use the same drivers (although probably different configuration for my card). That it happened on 2 different cards from different manufacturers does seem to suggest it may *not* be specifically related to the card, but rather an OS or TG2 issue. However, as I said, it does work for me and for others I have heard from as well, all with varying configurations.
I do know though that graphics drivers in Vista, particularly x64, have historically been problematic, particularly also in relation to OpenGL capability and compatibility. But again the fact that it works for me and others suggests there may be something else going on.
One thing that does occur to me is that your power supply is pretty low wattage even for low-end cards these days. Although it seems an odd thing to be responsible for these issues, in fact power supply problems account for a great number of unusual computer issues. The minimum recommend for even basic systems now is 350W I believe, and 420 or above is encouraged, so if your PSU is really 300W it's far short of where you probably want it to be. A 500W PSU from a reliable manufacturer is pretty cheap these days and, particularly with the RAM upgrade (which will also increase your power use), you probably need it at this point. I know my own quad core with 8GB of RAM and the 8800GTS needed a 600W PSU with specific power ratings to really work consistently stable. Something to consider...
- Oshyan
Quote from: Oshyan on October 13, 2008, 12:54:21 AM
I do know though that graphics drivers in Vista, particularly x64, have historically been problematic, particularly also in relation to OpenGL capability and compatibility. But again the fact that it works for me and others suggests there may be something else going on.
Yes, clearly. But, I don't know what that might be. I could try getting the 8800GT. However, I have read on forums that there are people having OpenGL troubles even on the 8800GT card when combined with Vista. So, it may simply be luck of the draw here that your configuration happens to work on Vista when others aren't quite so lucky. It's not as if I'm the only one complaining of this issue with Vista. One only needs to search Google to find myriads of forums with users complaining of OpenGL and video driver not responding issues.. both on Nvidia and ATI cards. I have even seen this issue span across operating systems and hardwares to affect even Mac OS X and Linux! This indicates that it may be a fundamental motherboard design/chipset issue in certain multicore motherboards. However, these non-Windows reported issues may be an anomaly and unrelated. But, I'm still not ruling out Vista entirely.
Quote from: Oshyan
One thing that does occur to me is that your power supply is pretty low wattage even for low-end cards these days. ... so if your PSU is really 300W it's far short of where you probably want it to be.
- Oshyan
Yes, however, I specifically chose this ATI card because the minimum recommended wattage for the card was listed as 300W. Yes, it's possible that my issue is related to low power issue. However if that were the case, I would expect to experience more problems with this system than I see. In fact, I have no other issues than this one OpenGL problem. I can watch DVDs, play iTunes, render images and do very CPU intensive things all without issues. On the other hand, I can trigger this video driver issue with a very specific set of circumstances which are exactly reproduceable. This indicates to me that it's more than likely a software issue instead of a hardware issue. I'm not ruling out hardware because I haven't changed everything possible, yet. Which software is at fault, I do not know. But, considering that Microsoft supplied an alleged fix to this EXACT problem in KB938194 (http://support.microsoft.com/kb/938194), I'm leaning towards Vista still being the root cause for this issue.
I should also point out that I've been having OpenGL crashing on this specific machine since even Vista 32 with 3GB of RAM (3 sticks). Granted, this crashing issue didn't affect Terragen 2 in Vista 32, but it does affect Poser. It still affects Poser in Vista 64. So. after moving to Vista 64, I'm now seeing this OpenGL crashing issue now affecting ALL OpenGL apps. It's no longer just Poser. Although, I haven't been able to get Poseray to crash OpenGL for some reason (unless it's using D3D). However, I can't help but think that these issues between Vista 32 and Vista 64 are related in some way.
Basically, no matter what I have done, I have not been able to get rid of the OpenGL crashing problem on this Quad Core system (either in Vista 32 or in Vista 64) by any method (either hardware or software). The only common hardware things left to change are the motherboard, CPUs and/or power supply. So, this means I am not ruling out a CPU bug in the 6600 quad core chip, a hardware bus or fundamental ASUS design problem, and I'm not ruling out a PS change. But, frankly, with this specific machine, I don't think changing the PS is going to substantially change the issue and will just serve to cost me more money and expend more time changing it.
Finally, I should point out that people running XP do not seem to run into this issue.
I see your points and it's clear you've done a lot of research on this which has led to reasonable suspicion of the OS and/or graphics drivers. I am inclined to think it may be a driver issue if anything. It's highly unlikely it is the motherboard or CPU, so if you do try one last hardware option I would try the PSU.
- Oshyan
Quote from: Oshyan on October 13, 2008, 01:43:12 AM
I see your points and it's clear you've done a lot of research on this which has led to reasonable suspicion of the OS and/or graphics drivers. I am inclined to think it may be a driver issue if anything. It's highly unlikely it is the motherboard or CPU, so if you do try one last hardware option I would try the PSU.
- Oshyan
Yes, but I find it hard to believe that both Nvidia and ATI would introduce the same driver bug that affects OpenGL under identical circumstances. The only way I can see this is if both manufacturers are using the same OpenGL code that is unmodified from the OpenGL consortium... I suppose that's possible. Of course, that still doesn't answer the question of why it doesn't affect your system.
Thanks.
Thought you might be interested... When the video driver stops responding, this is what I see in the problem report after it recovers:
Description:
A problem caused this program to stop interacting with Windows.
Problem signature:
Problem Event Name: AppHangB1
Application Name: tgd.exe
Application Version: 1.9.99.1
Application Timestamp: 4872d0aa
Hang Signature: df05
Hang Type: 0
OS Version: 6.0.6001.2.1.0.768.3
Locale ID: 1033
Additional Hang Signature 1: c0b029542fe376389c305cfa4e89e179
Additional Hang Signature 2: 7f34
Additional Hang Signature 3: 5abc92dd1f0408cfbb2d7b6ed2d5b208
Additional Hang Signature 4: df05
Additional Hang Signature 5: c0b029542fe376389c305cfa4e89e179
Additional Hang Signature 6: 7f34
Additional Hang Signature 7: 5abc92dd1f0408cfbb2d7b6ed2d5b208
Note, it always happens in the node area after I create about 3-10 successive drag highlight rectangles both upper left to bottom right and bottom left to upper right. I also drag the whole area around a few times as well for good measure. It's sure to hang the driver.