Large Tiff Heightfield Issues

Started by D.A. Bentley, October 31, 2017, 10:34:03 pm

Previous topic - Next topic

D.A. Bentley

October 31, 2017, 10:34:03 pm Last Edit: November 02, 2017, 10:32:30 pm by D.A. Bentley
I had some issues loading in very large 8-bit Grayscale Tiff files as heightfields.  I was loading in 40k x 40k maps, but then cropped down several times to get rid of some abnormalities forming at the bottom of my image/terrain.  Where there should have been just a flat black area (ocean) there was data forming in the area in a perfect horizontal bar.  I could crop my tiff image/heightmap smaller and eventually the bar went away.

Is this a known bug/limitation?

Oshyan

Strange, we're not aware of this problem. What is the source of the heightfields?

- Oshyan

D.A. Bentley

November 04, 2017, 06:31:29 pm #2 Last Edit: November 04, 2017, 06:43:44 pm by D.A. Bentley
Hi Oshyan,

I will assemble a test package for you to try out soon.  I can duplicate the issue, and take screen shots.  I can also share the tgd & tiff files with you to test.  It will be quite large though because of the 40x Tiff.  Should I send you a Dropbox link by email, or post the link here?  I'm on the latest "frontier build" by the way.

The heightfield is a single Tiff I assembled from 3 DEMs I downloaded from the USGS site.  I converted the IMG files using GDAL to 32-bit tiff files, and then in Photoshop downsampled to 16-Bit using HDR Toning.  I toned each of the three tiff files exactly the same way so that they would blend together nicely.  I combined them into one file which at this point was about 20k square, then I doubled the resolution to 40k before bringing the heightfield into World Machine for erosion.  So the heightfield went through many steps, but the final result was an export of a Tiff from World Machine. 

Thanks,

Derek

Oshyan

Thanks, that would be helpful. Dropbox link to support AT planetside.co.uk would be ideal.

It's unfortunate that the heightfield had to go through that much processing. It makes me wonder A: can't World Machine import at least GeoTIFF/32 bit TIFF? If so, use Global Mapper (or even Terragen) to export a single 16/32 bit heightfield and erode in Terragen. And B: if erosion is the main thing you want added to these heightfields, maybe now that Daniil Kamperov's "Classic Erosion" plugin is available you could use that instead and avoid all the resampling, etc. which is inevitably bad for height data. You could just import the DEM tiles directly into TG, they should align automatically if they're georeferenced (they will be from the USGS), and then just do the erosion on it and you're good. Just a new option to consider. :)

- Oshyan

D.A. Bentley

November 05, 2017, 12:14:44 am #4 Last Edit: November 05, 2017, 01:23:16 pm by D.A. Bentley
The reason I was going through that processing is because I manually added in some "fake bathymetry" to my merged DEM working file (in Photoshop) and wanted to use erosion to blend the terrain into the underwater area.  I couldn't find any bathymetry DEM for the area I'm working on (LA Coastline / Santa Monica Mountains), and i just needed some variation in the underwater terrain so the water would render more realistically. 

Also the USGS DEM I downloaded was in 1/3 arc-second DEM in IMG format from here: https://viewer.nationalmap.gov/basic/.  I tried using MicroDEM to convert to tiff with no luck, and then tried QGIS with Grass with some luck, but ended up using a command line batch convert in OSGeo4W Shell using GDAL.  I will look into Global Mapper, and I am planning on buying Daniil Kamperov's "Classic Erosion" soon.  I've just been scrambling this last week to finish a project that could land me a job so haven't had time to do everything properly.  ;)

I have attached two screenshots of what I am seeing, and I will send support a dropbox link to download the tgd scene and tiff heightfield.

OK thanks for the tips, and I'll try to learn as much as I can from the Wiki & Forum so I don't have to bug the support crew much.

Derek

Oshyan

Thanks for the files, we'll take a look.

- Oshyan

sboerner

Derek, have you seen this resource? NOAA's bathymetric data viewer:

https://maps.ngdc.noaa.gov/viewers/bathymetry/

It includes bathymetric DEMs for much of the North American coastline. The Los Angeles area is well covered and includes terrain along the coast. Maybe enough for what you need?

I used a few of the DEMs for an illustration a couple of years ago. Catalina Island is on the right, and I believe that those are the Santa Monica mountains on the left, behind the ship.

[attach=1]

Steve

D.A. Bentley

Hi Steve,

I did end up finding some DEM that extended into the Ocean contiguously (Integrated Coastal Relief Models).  I ended up downloading all the formats from the Grid Extract Tool, and settled on XYZ format.  It loads into Terragen without converting to anything, and it just works (although huge and slow).   

Grid Extract Tool:
https://maps.ngdc.noaa.gov/viewers/wcs-client/
OR big chunks:
https://www.ngdc.noaa.gov/mgg/coastal/grddas06/grddas06v2.htm

Actually I was just telling Oshyan about this resource, and thought the GeoTiff downloads didn't contain bathymetric data, but I was wrong.  Photoshop just has trouble with GeoTiff files.  Usually I can load a GeoTiff into Photoshop and use HDR Toning to Equalize the histogram to see all the data (in crunched form), but with the GeoTiff's with bathymetric data that doesn't work.  Everything below sea level just appears to be clipped (looks like a mask/all white).

I figured out that the bathymetric data was indeed in the GeoTiff by loading them into Global Mapper.  See attached images.

sboerner

Had to take another look at that website. It's much improved since I last used it. You can now download seamless GeoTIFFs in 1- and 3-arcsecond scale, at least around L.A. The download tool is here: https://maps.ngdc.noaa.gov/viewers/wcs-client/

Here is a very quick rendering of your area, color coded above and below sea level.


sboerner

Just saw your post! No need to tweak the GeoTiffs in Photoshop. Best not to touch them. Just load them into Terragen. They'll render just fine.

Matt

Quote from: D.A. Bentley on October 31, 2017, 10:34:03 pm
I had some issues loading in very large 8-bit Grayscale Tiff files as heightfields.  I was loading in 40k x 40k maps, but then cropped down several times to get rid of some abnormalities forming at the bottom of my image/terrain.


This is fixed in v4.1.17. Available now for Windows and Linux (check for updates in the app). The Mac version will follow shortly.

Matt
Just because milk is white doesn't mean that clouds are made of milk.

Oshyan

It's not totally surprising that Photoshop has trouble handling a GeoTIFF with bathymetry, I would guess the values underwater are negative and I don't think Photoshop can probably deal with negative values as its entire purpose is as an image editor for visible images, and in that context negative values don't really make sense.

- Oshyan

D.A. Bentley

Thanks Matt!  I'm not sure I will be using 8-bit heightfields anymore now that I know better, but it's nice to know this has been fixed.  I tested a 16-bit 40k x 40k Grayscale Tiff as a heightfield, and it worked flawlessly.  :)

Thanks again everyone,

Derek