Smooth by Mask

ejgodwin · October 21, 2016, 08:17:58 PM

Been AWOL for quite a while ... now I'm enjoying the new TG4. I love how fast the preview is!

I know it's been mentioned that TG isn't really designed for controlled placement of specific terrain elements (mountain range here, valley there, etc.), but I think I'm close to achieving a methodology that works. I started by eroding a simple grayscale heightfield image in World Machine, then exporting the result to a 16-bit image so I could tweak a few areas in Photoshop to get more precise control. Problem is, importing the image via heightfield load creates stairstepping in smoother parts of the terrain, due to the notorious "banding" effect of minuscule gradients. (See attachment -- I turned off fractal detail to emphasize the effect.) Changing the interpolation method from "Catmull-rom" to "Over smooth" doesn't really correct it, and besides, it takes away important detail from the mountains. Part of my problem, I think, is that I'm using a heightfield multiplier to achieve the proper elevation for the mountains, which of course magnifies the stairstepping effect.

I'm hoping to create a nice, gentle slope up the valley, perhaps by using an image map mask created in Photoshop. Unfortunately the heightfield smooth and erosion operators don't have a masking option. I'm sure there's a fairly simple way to do this via shaders, but the proper combination escapes me. The concepts behind them have always been tough for me to grasp.

Any help is appreciated -- including links to any tutorials I might have missed! Thanks.

Kadri · October 21, 2016, 08:45:49 PM

I think you know what you do, but still i want to ask if you worked in 16 bit in Photoshop?
If not, even if you export it as 16 bit you have lost quite a range of the image and the banding might come from there too?

ejgodwin · October 21, 2016, 08:57:45 PM

I tried 32-bit, but the effect was the same. Maybe I'm missing something in the export? I also made sure to convert to grayscale first, as exporting in full color wastes bit depth.

Kadri · October 21, 2016, 09:19:34 PM

I don't use WM. So can't say anything about that.
Without a file hard to say for me.
But the image you posted looks rather low resolution already.

Maybe others who use World Machine can help you.

Oshyan · October 21, 2016, 09:34:40 PM

You're seeing the results of an 8 bit conversion somewhere along the workflow. Not certain where. What is the resulting final image file format from Photoshop? TIF? Where are you "converting to grayscale", in Photoshop? If so, how are you doing it?

- Oshyan

ejgodwin · October 21, 2016, 09:53:48 PM

That's what I thought as well. The PSD file is in 16 bit grayscale, and I'm careful to export a png image with the same color depth. I originally created the image in 16-bit color, and I converted it to 16-bit grayscale. Not sure how that would be the cause of the problem. But when I zoom way in I can see the banding directly in Photoshop. My monitor is set to 32-bit, so I know it's not a perceptual thing.

Edit: I used the color picker tool in PS, and the two colors are only one bit apart: #202020 & #212121

Oshyan · October 21, 2016, 10:14:47 PM

Try saving it as a 16 bit TIFF...

- Oshyan

ejgodwin · October 21, 2016, 10:21:00 PM

That did it! Thanks, Oshyan.

Oshyan · October 21, 2016, 10:22:31 PM

No problem, glad it worked. I'm not sure if Photoshop exports 16 bit PNG. Or perhaps TG isn't handling 16 bit PNG properly?

- Oshyan

ejgodwin · October 21, 2016, 10:45:52 PM

Not sure, but I'm happy to use TIFF.

By the way, for anybody using Photoshop for heightfield images, you can't use the export function for TIFF files. You have to use "Save As ...", and make sure you check "As Copy" and uncheck "Layers" to reduce the image size. Not sure why they designed it that way, unless because TIFF has layering capabilities.

Dune · October 22, 2016, 01:50:56 AM

If layers is checked it might still have layers and it should be better to flatten them first, and also see if there are any alpha channels that you can get rid of as well, to lower file size. LZW compression works fine, btw. and is smaller again.

Oshyan · October 23, 2016, 04:31:46 PM

The "Export" function is intended only for web images, if I understand what you're referring to. TIFF is not a web-viewable format so that's why it's not supported. Exporting through the "Save for Web" dialog would also explain why even if you were in 16 bit and even if PNG can support 16 bit (I think it can), you were only getting 8 bit: because only 8 bit formats are supported in web browsers, and hence by Photoshop's export function.

- Oshyan