Hi all.
I'm trying to create some real-world terrain for the first time out (a LOT of learning today) and running into a roadblock.
I've downloaded a heightmap downloaded from USGS (https://apps.nationalmap.gov/downloader/#/) which I've imported. It works well!
The metadata places it on the TG planet (though not where I'd expect). I modified the "Lat long at Apex" in the planet node to bring the map to 0,0,0 to work with, as suggested by the documentation and these forums. All good so far.
Where things started to break down for me is trying to match up a color image map to the heightfield. The color image is a subset of the heightmap from USDA National Agriculture Imagery Program (NAIP), found in the same search as heightmap.
I downloaded the .jp2 image and imported into TG with the Geog Image Map shader.
Despite having metadata in the info tab and values visible in all the georeference fields, the image is placed with lower left corner at 0,0,0.
A few observations:
1. NESW corner latlong data seems to be in meters as opposed to degrees, like the heightfield
2. NESW corner latlong values are very big. They do not seem to be UTM data (no zone is able to be specified in TG, that I can see - how are these values calculated?)
3. I can switch between the Georeference and Position Explicitly radio buttons with no discernible change in the position of the image.
I *can* put values into the Position Explicitly fields to move the image map around, but I'd just be guessing to get a match with the correct position in relation to the heightfield.
....
There is SOOO much metadata for that .jp2 image.
It has latlong info, utm info...seemingly all that's needed to place it correctly, but I need help figuring out where to enter the data.
The Geo image map shader corner latlong data doesn't seem to be able to be modified, and I'm not sure how to convert the metadata from the web into the XZ positional data to place the image exactly where I need it.
Can someone pop these 2 images (links below) into a project and see if they match up for you?
Thank you!
Mike
heightfield:
https://prd-tnm.s3.amazonaws.com/StagedProducts/Elevation/13/TIFF/n47w123/USGS_13_n47w123_20210615.tif
color map:
https://prd-tnm.s3.amazonaws.com/StagedProducts/NAIP/wa_2015/46122/m_4612256_ne_10_1_20150729_20151123.jp2
meta data for .jp2 file:
https://www.sciencebase.gov/catalog/file/get/5828231fe4b01fad870f73c0?f=__disk__ca%2F85%2Fa8%2Fca85a84bdf1afd04315f2f9d5ee7da6c678bae6b&transform=1&allowOpen=true
Tried again with different images with the same results, with color image showing up at (0,0,0)
These threads seem to ask the same questions I am, but are from a while back.
https://planetside.co.uk/forums/index.php/topic,26700.msg266211.html
https://planetside.co.uk/forums/index.php/topic,25487.msg254816.html
The only fields which seem to have any effect on the placement of the .jp2 color image are Position Explicitly, which work even when the Position Explicitly radio button isn't activated (Georeferencing is ON). This makes me think Georeferencing has been disabled somehow?
I've included an image which has all of the data onscreen.
Help?!
Thanks all!
Mike
Have you tried checking "auto georeference from file"? When it's unchecked your map will be at 0,0,0 or whatever location settings are manually supplied.
Quote from: WAS on November 04, 2021, 12:20:22 AMHave you tried checking "auto georeference from file"? When it's unchecked your map will be at 0,0,0 or whatever location settings are manually supplied.
Hi WAS.
The Geog Image Map Shader node is behaving as though Position Explicitly is on no matter what buttons are clicked, or what data exists in the 4 corner slots.
Everything works as expected for the Geog Heightfield load node, including checking on/off "auto georeference from file", changing from the georeference radio button to the position explicitly button, etc.
I understand there is 3rd party code being used (GDAL) to ingest/calculate these things.
From the other forum posts, it isn't clear if issues with .jp2 files were ever sorted out?
M
I tried again.
I located an alternate NAIP color image source on USGS (Earth Explorer)
I found a GeoTiff (close to same position as .jp2). which loaded ~10x faster than the .jp2 format
The GeoTiff metadata was interpreted by the Geo Image Map Shader node in degrees, same as the heightfield (different than the .jp2)
However, the GeoTiff in the Geog Image Map Shader seems to behave the same way the .jp2 did, in that, it doesn't seem to recognize the georeferencing data. Booooo.
I decided to try and figure out the offset between the SW corners of the heightmap and colormap to place the image in the correct position.
I converted the lat and long (degrees) to x,z (meters) using this tool: https://www.cqsrg.org/tools/GCDistance/
This got me closer, but the colormap was still off.
I had to flip the image in Y (is this related? --> when importing the heightfield and I haven't yet moved the Planet, the heightmap shows up in the southern hemisphere)
After some time adjusting x,z values by hand, the map looks mostly correct.
So....I'm still looking for a solid lock of heightfield to colormap in TG4.
Anyone here gotten this to work successfully?
M
I can be honest, I've always had trouble with this as well, and this just adds to my confusion of how things are "supposed to work". At least in my head.
Maybe the coordinates systems aren't what they state they are? Maybe you can use one of the online conversion tools and convert them all to the same metric and system?
I had a look at the files in your first post. Opened them in TG and indeed the image map is off. I think this is because the image is in a different projection than the DEM/heightfield.
I also opened both files i QGIS (a free GIS program) and converted the image map to the same projection as the DEM. Then took the reprojected image file and loaded it as geog image map i TG. They now line up nicely as far as I can see.
I must admit that it´s not easy working with all these different projections and I´m not an expert in any way regarding these things. But perhaps this can help you in some way anyway.
Best regards
Robert
PS. The reprojected file is too big to attach here. Perhaps you can download QGIS and do the reprojection yourself. Or I can send it to you somehow if you want? DS
Quote from: WAS on November 04, 2021, 06:26:32 PMI can be honest, I've always had trouble with this as well, and this just adds to my confusion of how things are "supposed to work". At least in my head.
Maybe the coordinates systems aren't what they state they are? Maybe you can use one of the online conversion tools and convert them all to the same metric and system?
Something seems different in the way the Geog heighfield and Geog Image Map Shader handle data, but I can't be sure.
The GeoTiff color map in my latest test chooses degrees as it's projection system, just as the the heightfield does, but it's not translating to the correct spot in TG, whereas the heightfield seems to be. Also, having to flip the color image in Y confuses me.
I'm relatively new to TG, so not sure what may be bugs in the software and what is my user error.
I am learning a lot. ;)
I'll take Roberts suggestion and DL QGIS and try to convert images there to see if I can get that to work.
Quote from: Roberts on November 05, 2021, 07:05:07 AMI also opened both files i QGIS (a free GIS program) and converted the image map to the same projection as the DEM. Then took the reprojected image file and loaded it as geog image map i TG. They now line up nicely as far as I can see.
I'm going to try and replicate what you did, Roberts. Thank you for looking into this!
Well, I downloaded QGIS. It seems quite technical and very powerful (i.e. intimidating)
I opened both heightmap and colormap files in the app, and they appeared to be registered correctly to each other automatically. I then exported each to a new TIF file using identical projections.
In TG both of their units register as degrees, but my colormap still shows up at 0,0,0.
**Curiously, if I import the colormap using the Geog heightfield importer, it seemingly snaps into position.
Question:
Roberts, when you import your colormap using the Geog Image Map Shader, do the positional offset values you get (circled in green in your image) automatically populate for you? That has never happened for me in all of my testing so far.
Thanks for your help.
Yes, those values were automatically there (I did not enter them).
And yes, QGIS is intimidating. But you managed to use it successfully😉
Terp, do I understand you correctly that the image still show up in the wrong position in TG? It looks like it did in your screenshot.
Robert
Yes, the colormap is still at the origin (when imported with Geog Image Map Shader node)
The colormap imported via the Geog Image Map Shader seems to want positional data in order to offset to the correct spot.
The colormap imported via the Geog Heightfield Load node seems to move things to the correct spot (but we want it as an image, not HF)
I could try another QGIS export with a different projection system. I'm not sure why that would give different results, but I've never had positional data populate into the Geog Image Map Shader on import, ever.
There are still a lot of variables at play here.
Is the mismatch a QGIS projection export issue? Is it a TG4 build issue? (mac vs. pc? version? I'm using 4.5.56)
Another question: Robert, I imported your .tgd file - I can see and manipulate the heightfield in Terragen, even though the file path is pointed to your D: drive. How is that possible? Some kind of caching inside of TG?
Quote from: terp on November 05, 2021, 03:16:40 PMAnother question: Robert, I imported your .tgd file - I can see and manipulate the heightfield in Terragen, even though the file path is pointed to your D: drive. How is that possible? Some kind of caching inside of TG?
That seems strange indeed. I don't really know how TG works, but I would be surprised if the DEM is saved "inside" the tgd. But perhaps TG in some way finds the correct DEM because you also have it stored somewhere on your harddrive?
Looking at your screenshot it also seems strange that the DEM and the image map has the same corner coordinates. Since the image map covers a much smaller area, it should not have the same corner coordinates as the larger DEM. I will try to have a look at this, perhaps download the same image of the crater as you and see if I can match it with the DEM/heightfield. Not at the computer right now though.
Robert
Quote from: Roberts on November 05, 2021, 05:28:55 PMLooking at your screenshot it also seems strange that the DEM and the image map has the same corner coordinates. Since the image map covers a much smaller area, it should not have the same corner coordinates as the larger DEM. I will try to have a look at this, perhaps download the same image of the crater as you and see if I can match it with the DEM/heightfield. Not at the computer right now though.
Robert
Thanks again for your help, Robert!
If you're referrring to the screenshot in my latest post, the values are exactly the same because I'm importing the same colormap twice.
The screenshot shows the results of the same colormap used in a Geog heightfield Load node and Geog Image Shader Map.
I wanted to see if TG4 would move the image to the right spot if I used it as heightfield instead of an image.
The heightfield node seemingly moved the image to the correct spot, where the Geog Image Map Shader does not.
Quote from: terp on November 05, 2021, 06:16:38 PMThanks again for your help, Robert!
If you're referrring to the screenshot in my latest post, the values are exactly the same because I'm importing the same colormap twice.
The screenshot shows the results of the same colormap used in a Geog heightfield Load node and Geog Image Shader Map.
I wanted to see if TG4 would move the image to the right spot if I used it as heightfield instead of an image.
The heightfield node seemingly moved the image to the correct spot, where the Geog Image Map Shader does not.
Your welcome, glad if I can help.
Aha, then i understand why they are the same size. But I don't know why they end up in different positions. I'll try to have a look at this tomorrow!
Robert
By reprojected you mean changing the coordinate system / units right? That's what I thought the problem was. It always seems to be the issue with files I find, and I actually don't understand why corresponding files would have different coordinate ssytems when they're supposed to be bundled. Maybe there is software that handles colour differently then elevation data?
I believe there's an open bug from earlier this year with the geog image shader. It wasn't respecting the coordinates when I tried it in June. I don't know if that ever got resolved.
A bug would make sense, as The Geog Image Map Shader doesn't seem move anything via georeferencing, no matter what coordinate system (reprojection) I throw at it, while the Geog Heightfield Load always does.
Would be lovely to see that bit of existing code brought over from the heightfield load to the image map shader node.
How do I go about finding open bugs, or reporting what I think to be bugs?
QuoteThe Geog Image Map Shader node is behaving as though Position Explicitly is on no matter what buttons are clicked, or what data exists in the 4 corner slots.
QuoteYes, the colormap is still at the origin (when imported with Geog Image Map Shader node)
Can confirm that this is a bug in newer versions of TG (at least on Windows, not sure about Mac). It's been reported but I don't think it's been fixed yet.
Geopositioning does work as expected (AFAIK) on versions 4.5.13 and earlier.
Little frustrating that you already have a hard time understanding these file formats and their positions, to also know you've been just fighting a bug too with the software. Explains a lot though.
Well, I downloaded the same image from Earth explorer as you Terp (the one showing the west rim of the crater). When I load that into TG using the geog image map shader it lines up nicely with the underlying terrain. I didn´t have to reproject it first in QGIS. So TG seems to recognize and place it correctly. Perhaps this is because I´m using an older version of TG (4.3.23)
On the other hand, I´m having trouble restricting the image. I have to use a simple shape to mask the layer, otherwise the whole terrain outside the image turns black. I don´t know why, maybe I´m doing something wrong or there is supposed to be an accompanying alpha or something.
Anyway, I hope your project is coming along Terp, and let´s hope that Planetside can get rid of the possible bug in the more recent versions.
Robert
Quote from: WAS on November 06, 2021, 04:28:57 PMLittle frustrating that you already have a hard time understanding these file formats and their positions, to also know you've been just fighting a bug too with the software. Explains a lot though.
Yeah - I'm more of an artist-type...these worlds of data I've been been navigating would be enough for me to crack, given new/old image formats, projection conversions, coordinate systems, etc. Bugs add a whole new layer of complexity.
Quote from: Roberts on November 08, 2021, 06:48:19 AMSo TG seems to recognize and place it correctly. Perhaps this is because I´m using an older version of TG (4.3.23)
I bet that's it, Robert.
Quote from: Roberts on November 08, 2021, 06:48:19 AMOn the other hand, I´m having trouble restricting the image. I have to use a simple shape to mask the layer, otherwise the whole terrain outside the image turns black. I don´t know why, maybe I´m doing something wrong or there is supposed to be an accompanying alpha or something.
Strange behavior. Not having your version, it's harder for me to diagnose.
Have you tried changing the NoDATA (under the tab) to transparent or to some other color?
Maybe in TG's "compositing" of the layers, there are negative values being applied?
I see below-zero values messing stuff up all the time in Nuke over the years. ;)
Quote from: terp on November 08, 2021, 12:21:00 PMHave you tried changing the NoDATA (under the tab) to transparent or to some other color?
Yes, I have tried that as well as the different alpha/transparency setting under the effects tab. Unfortunately none seem to do the trick, all still makes the whole terrain surrounding the image map completely black.
But something struck me just now and that is that I connected the geog image map shader to the color input of a surface layer. Maybe it's better (or I'm supposed) to just use the geog image map on it's own? I'll try that when I'm back at the computer.
Robert
Quote from: Roberts on November 09, 2021, 12:31:47 PMYes, I have tried that as well as the different alpha/transparency setting under the effects tab. Unfortunately none seem to do the trick, all still makes the whole terrain surrounding the image map completely black.
But something struck me just now and that is that I connected the geog image map shader to the color input of a surface layer. Maybe it's better (or I'm supposed) to just use the geog image map on it's own? I'll try that when I'm back at the computer.
Hey Robert.
I'm running the shader directly under a compute terrain (or even the heightfield shader).
I ran into black pixels this morning too. I couldn't figure out a way around it in 4.5.56.
....
I did some more reading about QGIS this morning and found out how to connect the Google Satellite dataset as a layer within it.
So, now, I can import a heightfield (from USGS website) and they match in space.
I position the viewer window to the area I want, and export both heightmap and google satellite imagery using "Map Canvas Extent" option to make sure both are registered to the same 4 corners. This gets me two images which line up in TG4.5.56, BUT I got some black pixels (as you are getting) when there is empty data somewhere in my export region (Canvas Extent). In QGIS, I'm unable to effect NODATA parameters so far.
Again, learning a lot.
Quote from: terp on November 09, 2021, 03:50:07 PMThis gets me two images which line up in TG4.5.56, BUT I got some black pixels (as you are getting) when there is empty data somewhere in my export region (Canvas Extent).
Maybe try Unpremultiply in colour tab, or run the colour through a Clamp colour 0 shader (or even Clamp colour 0 1).
Glad you seem to have found a good solution terp! I don´t have a lot of experience with QGIS, but it has some really useful and fun tools for handling real world terrain and analysis.
Regarding the black pixels/NoData you get in QGIS, I know there are a ways to interpolate rasters in QGIS. I have the swedish language version, but if you look under "raster" and go down to "analysis", it should open a new list of tools. One of them should be something like "fill holes through interpolation". I have not done this myself though, but perhaps you can try that? If WAS suggestion doesn´t work of course.
I managed to get the geog image maps to display correctly in TG as well! :) As you can see in the screenshot there are several ways to connect it to a surface layer that works (no black void outside the image). So, the way I connected the geog image map shader before was wrong. Now I know better ;)
Robert
I've found the cause of the Geog Image Map Shader bug which was introduced in v4.5, and I've got it working correctly in development builds. We'll release a fix to 4.5 soon.
Thank you Matt, that's indeed good news!
Robert
Hey all - I was away on Holiday for a few days.
Robert: Glad to see you've got things working!
Matt: Thanks for tracking that down. I know you're a busy guy. I'll be on the lookout for your next version push.
Thanks again to you all for helping track down/puzzle out these issues, I appreciate your time.
As a followup, by adjusting the export output resolution of the color maps (via google maps) inside of QGIS, I am able to get some really helpful, higher-resolution maps onto the DEMs.
Bigger maps increase render time more than I was expecting, but it's good to know this option exists for whatever use-case you find this workflow helpful for.
The only problem with this kind of work is that you need to set the sun the way the shadows fall on satelite imagery.
Quote from: Dune on November 18, 2021, 02:02:17 AMThe only problem with this kind of work is that you need to set the sun the way the shadows fall on satelite imagery.
Yeah it's unfortunate the shadows are so hard. I've spent hours lightening shadows and clone tooling them out just for it to be "OK" and not really worth a large render.
Agreed guys.
Perhaps, as more maps become available from different sources, we'll be able to blend between them to mitigate shadows somewhat.
Otherwise, if you really need the map, it's time to find a good podcast/audiobook and cc/paint away. ;)