Planetside Software Forums

A new thread, which is hopefully easier to find for new users. I hope I covered it all and it's clear (and useful).

- I used the render camera to make an object from a surface (in this case a sphere with 2 water shaders to make a 'sea object').
- A new micro exporter should be set with no normals or UV's, or the outcome will be a disaster (setup for obj 2, tgout 002).
- All other nodes can be disabled; sun, enviro light, atmosphere, underwater textures, terrain, etc, so it's just the uncolored, but displaced sphere.
- MPD should be experimented with, as the obj can easily become huge. I set it at 0.25 with the render size intact. Hit render, and an obj will be made from the (in my case) cropped part. It only took 2mins in my case, resulting in a 31MB object.
- Import the obj, and attach the colors you need, but not the displacements to the object. Better also use a final transform shader set to final position to get the colors in world space. If it's water, don't forget to uncheck 'cast shadows' in the object rendering tab.
- Enable all disabled nodes for terrain, lights, atmo, and such.

Rendering will be much faster, especially with PT.

---

Good summary Ulco.

If you want, you could add that some objects will need "Force displacement" selected.

The holes in the object are interesting. I have not got so big problematic parts in my objects so far i remember.
The objects polygons are always not what you could say ideal, but this is quite different from what i got to this day.

If your example scene works in my old version the same way, i am curious what the reason is for those bad parts.

Ok! Nothing strange going on. Less detail, small object, more holes :)

So with higher detail a better object may be obtained, perhaps. I have to test whether the area with the holes renders fine.

Quote from: Dune on April 01, 2021, 07:38:49 AMSo with higher detail a better object may be obtained, perhaps. I have to test whether the area with the holes renders fine.

Yes. You get much better objects that are closer to the native Terragen look as you can see from the Lightwave obj look below.

This test  (Terragen v4.125) is with a 670 Mb obj file rendered in HD.

---

Ulco too tired now to test it. Haven't slept tonight. But that strange water crop render...
Just guessing but as it looks like you exported the water with normals checked, maybe the normals are all looking at the same angle.
So Terragen is calculating the object kinda like glass without any displacements.

It kinda looks interesting too.

Good to know more detail gives better objects. So the point is to find a sweet spot, not too big or it won't load, and not too small.

Yes, it looks interesting, and the only difference is in the way I exported (with normals and UV's checked). I guess it's the UV's that mess it up, not the normals. But you may also be right. I might have another go, but I'm getting tired about it too.

Thanks, Ulco, for summarizing all of this. I've been following the other thread with great interest. Thanks to Kadri and Was for your contributions as well.

One thing I haven't seen mentioned. When I used exported meshes for a project a couple years ago I found that Terragen exports many, many duplicate vertices. These can easily be eliminated using other software. I've generally used MeshLab (which is designed to clean up scanned meshes). It has a function to remove duplicates -- literally vertices at the same locations.

When I exported Ulco's watersphere (using his settings), TG gave me an obj file 237,533 KB in size consisting of 2,286,504 vertices. MeshLab removed 2,038,310 of these (no that is not a typo), reducing the total to 284,194 and exporting a new obj file 66,698 KB in size but identical to the original as far as I can tell. (View attached of the mesh in Blender.)

Blender's vertex merge (using the default distance of 0.0001 m) removed a few more, which makes sense because of the additional tolerance, yielding a result of 253,917 vertices. Blender's obj export function seems to be more efficient than MeshLab's, and saved out a new obj 35,700 KB in size.

MeshLab has a merge-by-distance function, too, but it's *very* slow and finicky so I didn't test it. Not sure if PoseRay has a similar function. Couldn't find it.

Anyway, this means that those huge obj exports we're getting from Terragen can be easily crunched down into files that are much more manageable.

Finally a question and a couple thoughts on this.

- Ulco, would you recommend this as a new workflow for water and other reflective surfaces?

- This might be less practical for large land surfaces because the resulting object isn't a planet, which means distribution and surface shaders should be set to "Use Y," correct? So if the original surface was generated from a large area, where the earth's curvature is noticeable, the resulting Y-keyed distributions on the object surface would be out of whack. (There may also be issues placing populations?)

- But for smaller foreground areas, or nonanimated water surfaces, this could be a lifesaver. When I disabled the planet (along with the atmosphere) in Ulco's scene and disabled all of the reflection/refraction settings in the water shader (leaving only displacements), TG exported the obj mesh very quickly, in just a few seconds. (I guess I could have disabled the sunlight as well.) Seems like this would be a very practical workflow for still images.

---

Quote from: sboerner on April 01, 2021, 11:08:30 AM...

- This might be less practical for large land surfaces because the resulting object isn't a planet, which means distribution and surface shaders should be set to "Use Y," correct? So if the original surface was generated from a large area, where the earth's curvature is noticeable, the resulting Y-keyed distributions on the object surface would be out of whack. (There may also be issues placing populations?)
...

I have that problem now actually in my animation. It is a big landscape.
The clouds are displaced over mountains. It changes from one side to the other side.
I wasn't aware at first. If you are aware of that you can act accordingly.

 As you can populate objects in any direction you want, that shouldn't be a problem in theory.
I wish that Matt fixed that rotated object bug for populations.
But if you don't rotate your main object (where the population will be sitting on) you should be fine.
As a workaround you could change the position, rotation the way you want in another software and load that obj.

Quote from: sboerner on April 01, 2021, 11:08:30 AM...
Anyway, this means that those huge obj exports we're getting from Terragen can be easily crunched down into files that are much more manageable.
...

Be careful. As you are a very good modeler you know this,
but when you automatically merge duplicated-bad vertices and polygons
you can get holes, alter the look more then you intended.
I begun to use direct Terragen exported objects because of this and because of editing time.
When you want to merge vertices, polygons in Lightwave a 3 Gb object can take a really awful long time.
Interestingly all those bad-duplicate vertices, polygons weren't much a problem from a render point of view for Terragen.
But can't say for sure that this will be so in all scenes.
I doubt that you will accept such garbage objects but just saying :)
In the past i decimated the objects to get smaller files but that produced here and there bad polygons too.
This might depend on the software you use too of course.
If the Terragen export was clean this would work very nicely
A Micro exporter optimised for better file size, direct decimating while exporting with more options would be great.
If it is relatively small like 1 Gb i clean the file. When it gets bigger i just use that as it is.

I don't care much but you can try rendering-exporting with only 1 thread (maximum thread setting not Windows).
From old posts here around it looks like this exports a cleaner file.
Takes longer of course.
I think there were some things related to the "Subdiv settings" part...
Adaptive etc...but don't remember which one did what if at all.
Have a look at those and see if it changes the export.

When I weld, and remove orphan vertices the object doesn't seem to change look. I havent tried directly as a render/import though. But indeed if disp is too fine you will get holes even with MPD2 i found with rocks. And unfortunately, even with welding, if there are holes you cant really subdivide as it will make those holes bigger. But if you need to subdivide you gotta weld TG objects as all the faces will shrink giving a weird patchwork mesh of holes.

With small files manual editing can be done.
But after a certain threshold it gets too hard. For me at least.

Quote from: Kadri on April 01, 2021, 01:27:39 PMWith small files manual editing can be done.
But after a certain threshold it gets too hard. For me at least.

Yeah like my geonosis stuff. Not sure where to from here right now. I think it may need to wait for better exporter. Cause if I could export the needed textures then a higher MPD may look fine just imported. Would need filesize issue fixed though too.

Also I think the water issue is the UV issue. Maybe dont use default from UVs and use procedural at all directions of normal. If its still there then must be normals.

Playing around with a recent scene to see how this might work with a real project. Exporting, cleaning, importing are not insignificant tasks and can take some time. If done once that's OK. If you make any edits to the water surface (for example) then you have to do it again. Takes care and planning.

Exporting at the original MPD of my test file (which is high, 0.8  ) creates a very large obj file with corresponding waits for cleaning, etc. So I'm experimenting to see how low that can be reduced and maintain quality. This could be a benefit, which would allow you to use a low MPD for low-frequency surfaces like water, while keeping a high MPD for high-frequency ground details in the final rendering.

I'm finding that MeshLab indeed does the best job fixing the duplicate vertex issue. I understand your warnings, Kadri. Meshlab counts vertices and faces. As an example, one very large file was fixed and the vertex count was reduced by 64,161,475 while only 76 faces were lost. So I think it's a pretty safe step to take, but I'll keep an eye on it. (I'm not doing any optimizing; the goal is to maintain the face count.)

Will have a test file to post soon.

So I played with MeshLab real quick on a rock object. It was 1.31gb, after running it through MeshLab I got a file that was 561mb.

I tried to make the mesh have really tight folds, and pocked it with rough high octave and high noise variation holes to try and break the mesh.

MeshLab seemed to do a great job, and no visible holes, and geometry looks decent enough. Textures don't align on the object anymore but oh well.

Obj - 42s
Cube - 2:21s

Quick note: However, there was an issue. The MeshLab object was no longer viewable in Poseray. It would load, but take a very long time, way longer than raw TG file. Then, it would be a blank preview, but when you select a material, the whole viewport goes red. So I tried zooming out... into infinity... and couldn't get the object to ever show. Lol

---

Quote from: sboerner on April 01, 2021, 11:08:30 AM- Ulco, would you recommend this as a new workflow for water and other reflective surfaces?

- This might be less practical for large land surfaces because the resulting object isn't a planet, which means distribution and surface shaders should be set to "Use Y," correct? So if the original surface was generated from a large area, where the earth's curvature is noticeable, the resulting Y-keyed distributions on the object surface would be out of whack. (There may also be issues placing populations?)

- But for smaller foreground areas, or nonanimated water surfaces, this could be a lifesaver. When I disabled the planet (along with the atmosphere) in Ulco's scene and disabled all of the reflection/refraction settings in the water shader (leaving only displacements), TG exported the obj mesh very quickly, in just a few seconds. (I guess I could have disabled the sunlight as well.) Seems like this would be a very practical workflow for still images.

I don't know if I would recommend this, but it's worth experimenting with at least. I think it depends on the scene, as you write. And mainly for reflective (small roughness), small displacement areas. With roughness above 0.2 it's not so slow, but under that, with tiny displacements, it's very slow.

I think it might be good for water too, especially if you raise subdivision for better detail underwater, but I haven't gone so far to test speed differences.
It also depends on your machine of course. If you have a fast machine and don't mind having a render cook while you're on another computer, or sleeping, I wouldn't worry.
I merely tested because I came upon a very slow area in a project that needs a big render, and didn't want my computer processing for days. If it's 24 hours or 8 hours, I prefer the latter plus making an object in a few minutes.

Regarding populating; if the original ground is still underneath (a cm lower or so), you can just use the object as an 'overlay' for the colors and reflections. Planet just needs the displacements then, no colors. Or a gradient from where the object stops. I haven't gone that far either. I still need to find out what an object exactly covers; if it's crossing with the original, or can be put just a fraction higher.

Jordan regarding subdividing, i am not sure that this is the best way especially if the exported object is a detailed big object already.
You are actually smoothing the object this way. You loose some hard-edges detail depending on the polygons etc.
If you did this to all your objects some of the changing detail could be because of this.

Regarding holes, if you still want to subdivide an object after you cleaned it, there is a "close holes" command in Lightwave for such problems.
I am sure Blender and others have something like that too. Then subdividing would be much better. But i still would caution about this as i said.

About "Subdividing" i should add i speak about the one that is using smoothing obviously.
You can subdivide without smoothing too. So not sure where you do this and in which way.

Quote from: WAS on April 01, 2021, 03:33:04 PM...
Quick note: However, there was an issue. The MeshLab object was no longer viewable in Poseray. It would load, but take a very long time, way longer than raw TG file. Then, it would be a blank preview, but when you select a material, the whole viewport goes red. So I tried zooming out... into infinity... and couldn't get the object to ever show. Lol

Inverted normals? As you can use double sided polygons it won't matter in Terragen.
If this is the problem there was an option for this i think in Poseray or at least you could flip them.

So here is a test I ran yesterday and overnight to see how practical this process might be with a real scene. (Spent a lot of time staring at the blue spinning wheel. :P  )

For me I'd say this is a typical water scene, about half water and half land surface/foliage. The original water surface is a lake object and extends about 220 meters from the camera. The rendering times given are for the full 2400x1000 pixel frame, which is cropped here.

I exported two versions of the water surface, one at MPD 0.8 and another at MPD 0.6.  The results for both aren't bad, though if you compare them carefully (not obvious here) there is some loss of resolution in both that affects the backlit details in the shadow areas. (Not sure why the lower MPD rendering time is slightly longer than the higher MPD time, but there it is.)

I was going to make more comparisons (MPD 0.4, etc.) but this was taking too much time and the conclusion seems obvious here.

Which is, for me, the savings in rendering time does not compensate for the time spent exporting and handling the mesh objects, which is about 25-30 minutes each on my machine. That means a real time savings of about an hour for each rendering. Again, for me, and for a scene this size, this is not significant. Plus, if you want to make *any* edits to the water surface, you have to go through the whole process again. And anyway I'd rather have the surface detail that gets lost.

So I would not recommend using the process for a scene like this. And never for a scene that is mostly water surface . . . the file sizes and processing times would go through the roof.

Maybe for scenes with smaller reflective areas, or more focused on foreground detail.

Interesting to give it a try, though.

---

That's a very nice comparison. It's hard to see any differences (especially if you don't know), so the water object works fine. If you don't need to reduce or clean an exported object it may be faster. In my case, making the actual object was done in minutes, and I imported right away.
Perhaps making the object is even faster if you disable literally everything but the lake displacement.
But I agree, it's not for every scene, but specifically for tediously plodding along areas, like rough, transparent water with upped subdiv settings and say rocks on bottom.

Nice comparison.

Curious are you sure that you did not accidentally added displacement to the imported object?

For an animation that time difference is important. But for an image i wouldn't bother too.

Quote from: undefinedPerhaps making the object is even faster if you disable literally everything but the lake displacement.

I made a duplicate of the original scene and disabled everything except the lake object (including lights, planet, and atmosphere), for exporting. Which took 5-10 minutes depending on the resolution.

Quote from: undefinedBut I agree, it's not for every scene, but specifically for tediously plodding along areas, like rough, transparent water with upped subdiv settings and say rocks on bottom.

Exactly. In the right context this could save a lot of time, especially with details where the water can be quickly finalized but you want to fine tune other details.


Or, even, make a water object to speed up test renders, then revert to the original for the final rendering. Lots of ways to work with this.

QuoteCurious are you sure that you did not accidentally added displacement to the imported object?

I used a copy of the original water shader but set everything on the Waves tab to zero. That should remove all the displacements, right?

When I exported the object I did the opposite - kept the Waves tab settings but zeroed out everything in the Reflections and Sub-surface tabs. Just to speed up the mesh export.

Quote from: sboerner on April 02, 2021, 01:19:05 PM

Quote from: undefinedCurious are you sure that you did not accidentally added displacement to the imported object?

I used a copy of the original water shader but set everything on the Waves tab to zero. That should remove all the displacements, right?

When I exported the object I did the opposite - kept the Waves tab settings but zeroed out everything in the Reflections and Sub-surface tabs. Just to speed up the mesh export.

Yes looks right.

Can't say for sure but from the other thread and this one, transparency gets a boost too
but it looks to me that the most big difference in render speed is with the reflection shader node
and especially with high displacements (many node etc.)  on landscapes.

Quote from: Kadri on April 02, 2021, 05:53:08 AMJordan regarding subdividing, i am not sure that this is the best way especially if the exported object is a detailed big object already.
You are actually smoothing the object this way. You loose some hard-edges detail depending on the polygons etc.
If you did this to all your objects some of the changing detail could be because of this.

Regarding holes, if you still want to subdivide an object after you cleaned it, there is a "close holes" command in Lightwave for such problems.
I am sure Blender and others have something like that too. Then subdividing would be much better. But i still would caution about this as i said.

I don't know what you mean by smoothing.

And actually subdividing lower MPD objs just smooths polygon edges. Ive done plenty of testing here and you can get game ready rocks at just a mb or two with MPD 0.1 and subdivide it. Sure it's not original displacement, but that's by design...

Same for geonosis disp. A full detail obj is over the OBJ threshold to begin with and would be a broken OBJ file. Its over 9gb. Im pretty sure most software would have trouble loading it, if it isnt broken itself at save time.

However like I said, I could use a smooth version of the terrain which isn't as large, and reapplying geonosis like displacement in Blender using same function formula (maybe even better since I have more noises).

---

As for this method Ive done days of testing and it seems highly redundant a method and you might as well use layer elements. Most these test scenes render longer than the whole process of importing SR reflections from standard to a full PT render and compositing real quick. For a muddy ground, again, there is no reason to go through all this work and its negligent imo. You aren't getting much benefits over whats available already, and you are creating larger projects on disk

Can't quite understand from your post. What are you using for subdividing?
And what is "geonosis"? Searched but couldn't find anything?

Geonsosis is a planet from Star Wars that I'm doing displacement like. That's what the spires terrain is, without the geonosis displacement. I was going to subdivide a bunch and try applying the disp back in TG, but the terrain has issues. So my next step is testing in Blender. Your trick of a "perspective" render it just odd, and wouldn't work for game terrain.

Just asked in which software you did use "subdivide"? Still not sure from your post.

Here we go gain...

I didn't say that the perspective obj method would be good for a game.
Do you know why? Because in games they do this on the fly (depends on the game) like Terragen does it.
Terragen uses what it made internally-dynamically and just exports it as a static frozen object (with the detail you choose).
The object you get with perspective is actually the same that Terragen uses (more detailed of course).
In export time you get a lower quality but it is just that same object actually.
If that is odd then you are using an odd software and most games too...at least some games.

Why are you still arguing i don't know i thought we got over this.
As i just wrote above that for an image i wouldn't even bother with this method.

Quote from: WAS on April 02, 2021, 01:48:46 PM...
And actually subdividing lower MPD objs just smooths polygon edges.
...

This not so easy as it looks as the terminology can be confusing in different software and even in a software itself.
Thus the reason i asked in which software you did this.

On the fly? No. They really don't unless you want bad performance.

Subdivision is done with whatever I do it with. Poseray, mesh lab, or blender, and the results with them don't seem any different unless doing more iterations then just 1. For that terrain it was all done in blender. 

And what you aren't getting is perspective models are just non-standard.... Anywhere. What TG is doing is irrelevant... That's TGs full render and the assets are unobtainable. But for games, and most other software and comping, tiles are the most common. Even when using heightmaps, they are tiles.

"What TG is doing is irrelevant."

What???

We are talking about exporting from TERRAGEN and bringing it back into TERRAGEN.
Every time i say something and make an example etc. you say that it is irrelevant or this and that.
Only what you say is relevant? ? ? ? ?

I said numerous times that if you don't plan accordingly your moves are limited with this method.
Sound this like it is OK for gaming? Did i in one place said that this is ok for gaming? NO!

I said the same that you have to use something like tiling here and in Facebook.
I don't know why and in which engine you want to use that but actually heighfields (and of course in tiles) are better for big landscapes for game engines like Unreal.

Still no scene example from you by the way. Keep nitpicking i have enough.

Quote from: Kadri"What TG is doing is irrelevant."

Quote from: KadriDo you know why? Because in games they do this on the fly (depends on the game) like Terragen does it

Games aren't generating terrain on the fly. They're baking out low poly objects, LODs, etc, or baking heightmaps to objects. Man, generating terrain on load or roam? That would be disasterous, as seen with sandbox games and their stability/performance. Games with dynamic terrain are terribly simplified.

I think too much is lost in translation for coherent conversation, is really the issue. Constant loops.

Quote from: WAS on April 02, 2021, 06:42:29 PM

Quote from: Kadri"What TG is doing is irrelevant."

Quote from: KadriDo you know why? Because in games they do this on the fly (depends on the game) like Terragen does it

Games aren't generating terrain on the fly. They're baking out low poly objects, LODs, etc, or baking heightmaps to objects. Man, generating terrain on load or roam? That would be disasterous, as seen with sandbox games and their stability/performance. Games with dynamic terrain are terribly simplified.

I think too much is lost in translation for coherent conversation, is really the issue. Constant loops.

Probably. As i was more talking about Terragen then games (in general only Terragen actually).
The part where games are morphing the LODs together in different ways was more what i was trying to say with "on the fly".
There are dynamic ways for this using it only where you need it then loading the full LOD etc.
Fully as Terragen is probably out of reach for some time at least.

Terragen works pretty well for OBJ terrains and  heightmaps. It's textures that are the issue. For example, can't export lateral textures vs normal when you're doing an ortho.

And old games did use a type of LOD for card-based projections of stuff, but that's pretty antiquated and rarer to see. But usually LOD is just level of detail, swapping objects for higher detail based on Z. And that's just a matter of loading the asset, fading out the current one, fading in the new, and unloading the other (if it's a sparse object, if it's recurring best to keep it loaded so it can swap in and out).

This is where better geometry would be nice though, as simplifying TG geometry is probably impossible. The geometry is all over the place and there is no uniform polygon mapping, so I don't think it would work well good. I tried MeshLab and it just disintegrated, even with a low threshold.

If it was a better optimized mesh, you could target just closeup quality, and simplify for LODs. But currently you'd need to render LODs based on MPD.

Quote from: WAS on April 02, 2021, 07:19:14 PMTerragen works pretty well for OBJ terrains and  heightmaps. It's textures that are the issue. For example, can't export lateral textures vs normal when you're doing an ortho.

And old games did use a type of LOD for card-based projections of stuff, but that's pretty antiquated and rarer to see. But usually LOD is just level of detail, swapping objects for higher detail based on Z. And that's just a matter of loading the asset, fading out the current one, fading in the new, and unloading the other (if it's a sparse object, if it's recurring best to keep it loaded so it can swap in and out).

Yes.

There is another method. Kinda similar but more what i was thinking about when i said "on the fly".
I have to search it...

Quote from: WAS on April 02, 2021, 07:22:32 PMThis is where better geometry would be nice though, as simplifying TG geometry is probably impossible. The geometry is all over the place and there is no uniform polygon mapping, so I don't think it would work well good. I tried MeshLab and it just disintegrated, even with a low threshold.

If it was a better optimized mesh, you could target just closeup quality, and simplify for LODs. But currently you'd need to render LODs based on MPD.

Yeah the mesh is really messy. That is one of the reason i actually don't care so much for cleaning it.

I tried a method for getting a really clean mesh. Hard to explain.
The problem was that for a big mesh it was toooo time consuming.
I tried it some years ago but i doubt that it would work now too as the files i use are much bigger too now.
But for small files it could maybe work.

I will search for a video for this too.

Quote from: Kadri on April 02, 2021, 07:24:40 PM

Quote from: WAS on April 02, 2021, 07:19:14 PMTerragen works pretty well for OBJ terrains and  heightmaps. It's textures that are the issue. For example, can't export lateral textures vs normal when you're doing an ortho.

And old games did use a type of LOD for card-based projections of stuff, but that's pretty antiquated and rarer to see. But usually LOD is just level of detail, swapping objects for higher detail based on Z. And that's just a matter of loading the asset, fading out the current one, fading in the new, and unloading the other (if it's a sparse object, if it's recurring best to keep it loaded so it can swap in and out).

Yes.

There is another method. Kinda similar but more what i was thinking about when i said "on the fly".
I have to search it...

This "Continuous Levels of Detail (CLOD)"...
"https://en.wikipedia.org/wiki/Level_of_detail_(computer_graphics)

Quote from: Kadri on April 02, 2021, 07:29:34 PMI tried a method for getting a really clean mesh. Hard to explain.
The problem was that for a big mesh it was toooo time consuming.
I tried it some years ago but i doubt that it would work now too as the files i use are much bigger too now.
But for small files it could maybe work.

I will search for a video for this too.

This method:
https://www.youtube.com/watch?v=MwoPTl5ykiY
Go to 2:20.
But as i said not practical for high resolution terrain objects.
And overhangs might still be a problem with this even if it works.

I tried 3DCoat 1-2 years or so ago for clean meshes but it couldn't load those big files.
Maybe it is better now. I will try.
I don't know if Zbrush is better in this way.

You're hijacking my thread 8)  I started this one anew, to hopefully keep it short and usable for users, with a minimum of 'clutter' ;)

At least it looks like there is quite an activity here around ;D

;D  Indeed.