Planetside Software Forums

General => Terragen Discussion => Topic started by: chrisshort on March 09, 2015, 04:25:48 PM

Title: Need help with using The National Map website
Post by: chrisshort on March 09, 2015, 04:25:48 PM
I'm sure this may have been touched on but I couldn't find it in my searches.  I'm trying to download both a DEM of a particular location (in the USA) along with the same ortho color map to bring into Terragen.  I have been able to download this: USGS NED n39w106 1/3 arc-second 2013 1 x 1 degree ArcGrid and bring it into Terragen. My main problem is getting the appropriate color map that is geosynched to this elevation. They give me six 10:1 Compressed NAIP jp2 files, which seemed to be in grayscale.  The map they have to select the area is in color and with good resolution (when I zoom in) but that isn't what I can download.

Is there anyone who has used this website and can give me a "Best Practices" as to what files to download etc.  Any insight would be helpful....
Title: Re: Need help with using The National Map website
Post by: bigben on March 09, 2015, 05:06:14 PM
While it's not an answer about this particular site, it may help.  I use GlobalMapper for my map data and it includes direct connection to those data sources that have publicly accessible APIs.  If QGIS (which is free) has the same data connections, this is probably the easiest way as you can then just navigate to an area and download whatever data you need for the area you want, all at exactly the same coverage. You can also down sample the data to meet your needs at the same time e.g. It's possible to preview the entire US 10m DEM dataset and down load it at 500m/px as a single file. No worrying about whether you have a particular tile or not.
Title: Re: Need help with using The National Map website
Post by: archonforest on March 09, 2015, 05:18:22 PM
I managed to get data out of them based on this write up:
First, the current URL is: http://viewer.nationalmap.gov/viewer/

Go there and find your area of interest. Navigation controls are similar to Google Maps. Ideally it should not be a really huge area.

Now click Download Data in the upper-right. You should then get a little pop-up window with some options for choosing what to download. I like "Draw and download by bounding box', but any of them are potentially useful, and in particular if you have zoomed all the way in to only show the *exact* area you want, then "Download by current map extent" may be good.

Once you select your download area, another window should pop up showing download data options. Check Elevation (second from bottom), and any other data you want. Then press Next.

You will then get a list of available data. There are usually a lot of similar options. The two major differentiators are 1: resolution (under the Resolution column), which is generally measured in arc-seconds, the smaller the number/fraction, the higher the resolution and bigger the files for equivalent area, and 2: format (format column), where you'll find there are lots of the exact same data (same resolution) just in different formats such as ArcGrid and IMG. The good news is Terragen 3 supports almost every format they offer, particularly in terms of heightfield/elevation data, so it doesn't really matter what you choose. Just find the data you want at the resolution you need (usually the highest available, unless file/download size is a concern for you), check it in the box to the left of that item, then click Next at the bottom when you've selected all you want (downloading multiple resolutions is not necessarily bad, but no need to download multiple formats of the same data).

Once selected your data option(s) should show up in the Cart to the left. Then press Checkout. You'll need to enter an email address, don't worry they won't spam you.

You should then get an email with download links.
Title: Re: Need help with using The National Map website
Post by: chrisshort on March 09, 2015, 10:46:40 PM
Thanks. I don't have an issue getting the elevation, it is the color map I'm not sure what to select. It only seemed to give me that grayscale version while the viewer image was what I would have liked (Especially if it has the geosynch attached....
I'll look at the two programs mentioned as well...
Title: Re: Need help with using The National Map website
Post by: mash on March 10, 2015, 01:00:41 AM
You want to select "Orthoimagery" and then select 10:1 Compressed NAIP 1/1
You may also try
http://earthexplorer.usgs.gov/
The selction there is much easier to find.
Aerial Imagery > High resolution Orthoimagery
Just make sure you have an account set up and your logged in before you select your data.
The site has a bug where it crashes if you select data then try to log in.


Title: Re: Need help with using The National Map website
Post by: bigben on March 10, 2015, 05:54:13 AM
NAIP, NED and Topo are all available as WMS for GIS applications.  If you're looking at doing a lot of real world stuff it really does make it easy getting the data.  Just browse to the area you want and then download the data at whatever resolution you need.
Title: Re: Need help with using The National Map website
Post by: chrisshort on March 10, 2015, 08:27:53 AM
Quote from: mash on March 10, 2015, 01:00:41 AM
You want to select "Orthoimagery" and then select 10:1 Compressed NAIP 1/1
You may also try
http://earthexplorer.usgs.gov/
The selction there is much easier to find.
Aerial Imagery > High resolution Orthoimagery
Just make sure you have an account set up and your logged in before you select your data.
The site has a bug where it crashes if you select data then try to log in.

I did download the  10:1 Compressed NAIP, but as I mentioned, it opened up in photoshop as a grayscale image. Was I supposed to do something else with it?
I'll examine your other website option. Thanks.....
Title: Re: Need help with using The National Map website
Post by: chrisshort on March 10, 2015, 08:31:29 AM
Quote from: bigben on March 10, 2015, 05:54:13 AM
NAIP, NED and Topo are all available as WMS for GIS applications.  If you're looking at doing a lot of real world stuff it really does make it easy getting the data.  Just browse to the area you want and then download the data at whatever resolution you need.
Thanks!  I see your NAIP is in color. I'm not sure why the one I have opens up in photoshop in grayscale?
Title: Re: Need help with using The National Map website
Post by: Ashley on March 10, 2015, 08:17:50 PM
All the suggestions so far are great, I'm adding this link since I find it the most direct resource for DEMs.
I had similar confusion when I started using DEMs, the endless links, the rabbit hole is deep.

http://www.imagico.de/map/demsearch.php

To use these DEMs marquee a rectangular selection, then hit search (bottom left of map)
A list of lat long co-ordinates will be listed below and you simply download what you want.
In TG go to terrain tab and select load multiple DEMs.
To line it up in viewport copy the lat long co-ord from any DEM load node and paste it to the planet objects lat long apex.

The trick is to know what the lat, long is of the geo ref you're looking for, wikki is good for that info.

Hope that helps and I'm glad to see others are posting their resources, DEMs is addictive. 
Title: Re: Need help with using The National Map website
Post by: bobbystahr on March 10, 2015, 08:29:58 PM
Quote from: Ashley on March 10, 2015, 08:17:50 PM
All the suggestions so far are great, I'm adding this link since I find it the most direct resource for DEMs.
I had similar confusion when I started using DEMs, the endless links, the rabbit hole is deep.

http://www.imagico.de/map/demsearch.php

To use these DEMs marquee a rectangular selection, then hit search (bottom left of map)
A list of lat long co-ordinates will be listed below and you simply download what you want.
In TG go to terrain tab and select load multiple DEMs.
To line it up in viewport copy the lat long co-ord from any DEM load node and paste it to the planet objects lat long apex.

The trick is to know what the lat, long is of the geo ref you're looking for, wikki is good for that info.

Hope that helps and I'm glad to see others are posting their resources, DEMs is addictive. 

These are .hgt files that downoaded...does TG3 read these?
Title: Re: Need help with using The National Map website
Post by: Ashley on March 10, 2015, 11:03:37 PM
These are .hgt files that downoaded...does TG3 read these?

Yep!
Title: Re: Need help with using The National Map website
Post by: bobbystahr on March 11, 2015, 01:02:31 AM
Quote from: Ashley on March 10, 2015, 11:03:37 PM
These are .hgt files that downoaded...does TG3 read these?

Yep!

coolio....
I downloaded the area I live in and when my current render is done I shall experiment...
Title: Re: Need help with using The National Map website
Post by: Oshyan on March 11, 2015, 01:34:10 AM
For color images, you want to go to the Shaders tab, click Add Layer, and choose Load Multiple Geog Image Map Shaders. Then you should be able to load all your JP2s. I would not trust Photoshop to represent them properly, they may indeed be grayscale, but perhaps not...

- Oshyan
Title: Re: Need help with using The National Map website
Post by: bobbystahr on March 11, 2015, 01:51:24 AM
Quote from: Oshyan on March 11, 2015, 01:34:10 AM
For color images, you want to go to the Shaders tab, click Add Layer, and choose Load Multiple Geog Image Map Shaders. Then you should be able to load all your JP2s. I would not trust Photoshop to represent them properly, they may indeed be grayscale, but perhaps not...

- Oshyan

also coolio....so, .hgt are as well .jp2 ?
Title: Re: Need help with using The National Map website
Post by: Oshyan on March 11, 2015, 02:10:18 AM
No, HGT should be height as far as I know, not orthoimagery. But the original discussion seemed entirely focused on aerial/satellite imagery, the height data seemed to be coming in fine. So I was answering the original question. *Your* data download is probably height and should be loaded on the Terrain tab with Add Terrain -> Heightfield Load (use Load Multiple DEMs if you have more than one, though I think you're using free version and this option may not be available).

- Oshyan
Title: Re: Need help with using The National Map website
Post by: bigben on March 11, 2015, 08:08:35 AM
A very rough walkthrough.... QGIS example using NAIP.

* Get QGIS (http://qgis.org/ (http://qgis.org/)) and start the desktop program
* Layers > Add layer > Add WMS/WMTS layer
* Copy this URL into the URL field: http://raster.nationalmap.gov/ArcGIS/services/Orthoimagery/USGS_EROS_Ortho_NAIP/ImageServer/WMSServer
[attach=1]
* Connect to the service, select the layer, add it and close the dialogue window
[attach=2]
* Zoom to the area you want (or use the coordinate/scale boxes at the bottom of the screen)
* Right click on the layer name and select "Save As"
[attach=3]
  * Enter a directory to save the image(s)
  * Click on Map Extent button to save just the area displayed
  * Enter the horizontal and vertical resolution (decimal less than 1)  Use the same number for both so that you get square pixels. The total image dimensions will appear in the "columns" below.  This gives you control over the image resolution so you don't have to download more than you need.

Hit the OK button and it will get 2000x2000 pixel geotiff tiles and put them in the specified directory. 
[attach=4]

If you want the raw resolution data, Enter the resolution as 1/display scale. eg. if the scale is 1:10,000 use a resolution of 0.00001.

No messy searching for tiles, multiple download links, registration/logins. Get what you want at exactly the resolution you want. :)
Title: Re: Need help with using The National Map website
Post by: Ashley on March 11, 2015, 06:33:35 PM
@bigben
That looks awesome, thanks for outlining the process :-)
Title: Re: Need help with using The National Map website
Post by: chrisshort on March 11, 2015, 07:26:46 PM
Thanks! A walk through is just what the Dr. ordered. I do have That program to try.

Thanks Also to Oshyan for explaining how to use the NAIP in Terragen directly. This needs to be documented somewhere as it isn't obvious that this method exists....

Thanks to all!
Title: Re: Need help with using The National Map website
Post by: bobbystahr on March 11, 2015, 09:13:20 PM
Quote from: Oshyan on March 11, 2015, 02:10:18 AM
No, HGT should be height as far as I know, not orthoimagery. But the original discussion seemed entirely focused on aerial/satellite imagery, the height data seemed to be coming in fine. So I was answering the original question. *Your* data download is probably height and should be loaded on the Terrain tab with Add Terrain -> Heightfield Load (use Load Multiple DEMs if you have more than one, though I think you're using free version and this option may not be available).

- Oshyan

That's the route I took Oshyan, but they all loaded at 0,0,0 so I missed a step obviously...will read more now I know they load fine. Thanks.
Title: Re: Need help with using The National Map website
Post by: chrisshort on March 12, 2015, 09:40:44 AM
Quote from: Oshyan on March 11, 2015, 01:34:10 AM
For color images, you want to go to the Shaders tab, click Add Layer, and choose Load Multiple Geog Image Map Shaders. Then you should be able to load all your JP2s. I would not trust Photoshop to represent them properly, they may indeed be grayscale, but perhaps not...

- Oshyan
OK, Attempted to load the NAIP files. Here is the current situation. What was downloaded based on my selection was two DEM files and 6 NAIP files (1 through 6).  I load both DEMS and all 6 NAIP files. They all have the geo referencing on.  I then change the planet apex to one of the NW corners to have the dem oriented in my view (Is is correct?).

I don't see the mapping yet so I check the coordinates of the images and they don't match the proper location. If I plug in each corner info of one of the dems (Because that is the only info I have)  map 1 shows up over that particular DEM (which is expected) but I'm sure the 6 images needed to be sized to cover the two dems and I don't know where to get the info of each corner of each image. The email of the download only gave me "Current Extent/(-105.125, 38.825), (-105.016, 38.92)" for each of the six images while the dem's seem to cover -106 through -104 and 39 through 38.

Is it possible the NAIP files don't have the Geo referencing attached and I need to manually get the coordinates for each segment?  Bummer if that is true...
Title: Re: Need help with using The National Map website
Post by: Oshyan on March 12, 2015, 05:41:34 PM
It sounds like either the data does not have georeferencing information embedded in it, or TG is not reading it right. You would need to confirm the existence and accuracy of any georef data in it by using another app like Global Mapper. Perhaps BigBen can have a look?

- Oshyan
Title: Re: Need help with using The National Map website
Post by: bigben on March 13, 2015, 07:04:46 PM
JPEG, JP2 doesn't support embedded coordinates. This info is usually included in a metadata file with the image, although some sites just provide the image. In those cases you'd need to record the corner coordinates and projection when acquiring the data. If you have this info you have to manually georeference the image in TG.

Geotiff is by far the easiest format to use for TG. QGIS is a bit... OK, a LOT... quirky if you're not used to all of the techy GIS stuff, but the workflow I described above is relatively easy and gives you geotiff.  I tried adding similar WMS services for elevation that appear in GlobalMapper, but it looks like GlobalMapper has a lot of hidden smarts to make accessing the data really easy. I'll make a tutorial if I can figure out some of my data workflows with QGIS in a reasonable simple way.
Title: Re: Need help with using The National Map website
Post by: chrisshort on March 14, 2015, 04:03:06 PM
Quote from: bigben on March 13, 2015, 07:04:46 PM
JPEG, JP2 doesn't support embedded coordinates. This info is usually included in a metadata file with the image, although some sites just provide the image. In those cases you'd need to record the corner coordinates and projection when acquiring the data. If you have this info you have to manually georeference the image in TG.

Geotiff is by far the easiest format to use for TG. QGIS is a bit... OK, a LOT... quirky if you're not used to all of the techy GIS stuff, but the workflow I described above is relatively easy and gives you geotiff.  I tried adding similar WMS services for elevation that appear in GlobalMapper, but it looks like GlobalMapper has a lot of hidden smarts to make accessing the data really easy. I'll make a tutorial if I can figure out some of my data workflows with QGIS in a reasonable simple way.

Thanks for the input! In the case of the NAIP files, Since it was broken up into 6 parts and only give one overall coordinates, how would I know how to do the coordinates for the 6?
Title: Re: Need help with using The National Map website
Post by: bigben on March 15, 2015, 05:22:50 AM
If it's lat/long projection (safest bet to start with anyway), there'll be a direct correlation of pixels to coordinates. If the images are not all the same size, just calculate the size in degrees of 1 pixel (lat/long difference divided by total image height/width) and then go from there.