Shape files - how, what, where, why

[jo]

Hi,

I hijacked the "Accessing population data" thread a little with a bit of a tangent about Shape files. I thought I would start this thread to solicit comments about Shape file support in TG2.

For those who don't know, Shape files are a vector based Geographic Information Systems (GIS) file format which is commonly used to define features and areas in the real world.

I'm going to list some things I'm interested in hearing about. I have plenty of ideas of my own but I'd like to hear yours. Feel free to add whatever you like though.

Are you interested in using Shape files in TG2?
Are you already using Shape files but importing them some other way (as rasters for example)?
What are you interested in using Shape files for?
What do you feel is the basic level of functionality you'd need to effectively work with Shape files in TG2?
How important is it for you to be able access attributes in TG2 itself, or would you be happy with setting up data for import in a more specialised GIS app?
The same question as above but for operations like buffering or performing queries?
The same question as above but with regard to managing projections i.e. would you expect TG2 to work with all projections thrown at it or would you be happy to set up a projection in a more specialised app for importing as something TG2 understands?

As I said above, feel free to mention what you wish.

Regards,

Jo

[Henry Blewer]

I have not used the shape shader except for experimenting. I have not found anything which lets me know how to use them effectively.
It would be nice to get/have a way to add populations from external data. In Vista Pro, later World Construction set I used image maps for this. It is not hard to do this now. I created the image maps from aerial survey images and masking.

My attempts to download GIS data from USGS have been so long with the wait times, I gave up.

[dandelO]

http://en.wikipedia.org/wiki/Shapefile

[FrankB]

Hi Jo,

interesting thought, using shapefiles in TG2.
A practical use case I can think of is using shapefiles as masks for object placement, procedural displacements, etc... to mimick places in the real word.

Now, I don't recall if that is a feature that anyone ever asked for yet. I have followed some, but not all, of the "your top 5 feature requests" thread on this site and I don't recall anyone mentioned shapefiles.

Nonetheless this might be a great idea! One big question I have about this though is: where do I get shapefiles from? If it's some proprietary format and the data for it not generally available, or awfully expensive... I would be less interested, you see?

Best regards,
Frank

[freelancah]

I suppose some of these could be used if there would be support ? https://isis.astrogeology.usgs.gov/IsisSupport/index.php?topic=1166.0

[cyphyr]

The difficulty with using procedurals is that they only really work in completely wild areas. As soon as we create anything that is influanced by Man an element of order (or at least non-randomness) 9is added to our scenes. For example using a Terragen populator to make a city or town will me highly problematic with multiple overlaps, building facing wrong directions and not following roads etc. Whereas building en entire town has its own set of problems, file size not being the least. A shape file solution would be ideal (if it could be "shoe-horned" in the right way).
If we could define local density, rotation and path following attributes that "grey" area between random procedural and ordered/man made/influenced (fall off between the two would be good too) then terragen would be capable of images no other software could produce.

Richard

[RArcher]

Hi Jo, Glad to see a dedicated topic to this.

I currently use shapefiles fairly extensively when I am working with real world data (in particular Canadian data, but more on that later).  My current process of using the shapefiles is converting each layer that I want to use in GlobalMapper to a black/white raster image that I then use as a mask in TG.  It currently takes quite a few steps to get the data organized and imported and properly aligned.  For example:

1. Find data online or from an archive
2. Find the layer required and open it in GlobalMapper
3. Adjust projection, datum etc.
4. Set the area feature types to be a black/white mask
5. Export the data as a high-res image mask
6. Import my terrain into TG 0.9 to find out what size in meters it actually is
7. Import the image mask into TG2 and then fight with scaling and aligning the masks properly
7a.  Normally I export all my image maps as the same size as the terrain I am working on just to remove the headache of trying to align each mask.

One way to solve 90% of my issues would simply be to allow us to georeference image maps (which Matt has previously said he wants to do eventually).  This way using any size or location of image mask would automatically position itself on the georeferenced terrain.

Here is a short list of what I would like to see included in TG2 that would make my life easier:

- Allow georeferencing of image maps
- Display the size (in meters) of imported terrains so I don't need to open them in TG 0.9 to get this information
- Lat / Long displayed in 3D viewport when working with georefereced terrains
- Lat / Long option for positioning (individual) objects ie. traffic signs, light poles, trees
- Import of UTM based projections (this is mostly selfish reasons as all the Canadian data I use is in UTM format)

Directly opening shapefiles in TG2 seems like it would be a nice thing to be able to do, but I don't know if you really want to go down the path of having to build a GIS application into TG2.  There are already a number of different programs of varying complexity (GlobalMapper, ARCview, MapInfo) that handle this function quite well and in the case of ARCview are already extremely complex pieces of software.  If anything I would rather see TG2 be able to use the data provided from other sources easier rather than have TG2 do everything internally.

Now as to your questions specifically:

1. Yes, I am interested in using shape files (or at least the shape file data) in TG2
2. Yes, I import shape files as image maps
3. I use shape files for rough positioning of feature types (forests, water bodies, glaciers, grasslands, marsh, etc.)
4. Georeferenced image maps, Lat/long display, and positioning via Lat/Long would give me a great deal of power and flexibility for using the type of data mentioned above. 
5. Attributes are not of great importance to me, there are far better specialized applications for this
6. See above
7. I would like to see UTM as a projection option but that is simply because I am lazy and don't like having to convert the data I use  - There are so many different projections in use that adding them "all" would be a nightmare.

Here is a short tutorial I wrote back in 2008 about working with the Canadian data and shape files.  Some of the information is certainly out of data as we were not able to georeference terrains when I wrote the guide, but it should explain to anyone not familiar with shape files what is possible in a simplistic fashion:  http://www.archer-designs.com/tutorials/terragen2/working-with-gis-data/index.html

More thoughts to follow as they come to me...

[jo]

Hi,

This a copy of a message Eikers posted to another thread, which I'm posting here with permission:

I would love to use both point data for locating things (whether that is individual trees, or traffic signs, light poles or whatever) as well as area features for tree coverages, marsh land, etc. etc.

Hi RArcher,
I could not have said it better!
Points are very versatile and could perhaps be used as a support when drawing masks as well? But yes, using area features directly would be better.

Hi Jo,
I am interested in visualizing real world nature data, including measured trees. This can be done in other applications, but it always looks very stylistic and the closer you get to the trees in a scene the emptier it becomes. So using TG2 to fill in the blanks in the geographic info by creating more detail with terrain displacements and shader effects as well as plant populations and other objects would greatly improve the visual quality and versatility of such scenes.

Another way of using shape files (or GIS data in general) would be to symbolize them as more traditional map icons inside a landscape scene. Say you have a bunch of campsites. These could be shown as a population of a "campsite looking 3D object" marking their location. An advanced map or map enabled 3D landscape don't necessarily need more GIS input than a coarse terrain model and a water mask. With the procedural powers in TG2 I simply think there are many creative possibilities when merging real world data with photorealism and procedural detailing.

Erik.

[jo]

Hi Frank,

Nonetheless this might be a great idea! One big question I have about this though is: where do I get shapefiles from? If it's some proprietary format and the data for it not generally available, or awfully expensive... I would be less interested, you see?

Shape files (I should really be calling them Shapefiles) are available from many sources, and are certainly available for free. As an example you can download the Shape files used to create the NZ 1:50,000 topographic maps for free, which gives you access to information such as areas of vegetation, lines of roads, point features, fence lines etc. etc.

It's an open format which was created by ESRI, the biggest GIS software company, and is supported by lots of other apps.

Regards,

Jo

[Eikers]

I'll just skip the small talk and go fill in Jo's questionnaire.

Are you interested in using Shape files in TG2?
Yes, there are much data stored as Shapefiles or as vector GIS data in general, that are better visualized in 3D rather than conventional maps. Shapefiles are often the exchange and work format. These data sets are also gradually becoming more accessible to the general public as more nations migrate towards the geodata policies of especially Canada and USA. Apart from general land cover many engineering / consultant / landscaping companies make detailed plans in some vector format because of the precision and identity benefits. Esthetics play an increasingly important role in todays decision making processes. In brief: Some people read maps well but everybody read a good illustration even more so a photo-realistic rendering complete with "special effects" such as fog, snow or grime. It has to be sufficiently easy to import enough map elements to recognize the project so that one feels confident that TG2 can finish the task. For a newbie to TG2 this is like going abroad and find that people understand some of your language; oh they know "beer" here too - this will be just fine. The key word for the GIS world is geo-referencing.

Are you already using Shape files but importing them some other way (as rasters for example)?
I have tried to import vector data to use as masks using the excellent tutorial from Archer Designs. I found that there were quite a bit of conversion and projection issues, but where RArcher says this represented 90% of his work flow, it was 400% of mine, being a first timer. So I feel the threshold for using GIS data in TG2 is too high. This surely has to do with my background working with GIS and being used to seeing the map layers just snap into place as they are imported, and having to reproject or align anything is more and more rare.

What are you interested in using Shape files for?
There are at least three areas I'd like to use shapefiles for. Firstly for generating masks for shaders. Detailed data sets on land cover and land use exists in various forms in many countries. The sets are often represented as polygons because of storage and accuracy advantages. In the real world of course, areas are not categorically divided like that, so land cover data need to have fuzziness added to look credible in an 3D rendering. A typical strong TG2 feature.   
The second subject is shape files as input for object population. As I understand this TG2 objects are positioned with a single coordinate and can have rotation and scale applied to it.  The shape file representation is a point (2D or 3D) with attributes that can hold information on rotation and scale or something to scale by. To me that seems like a perfect match. Electricity masts, trees and more have been mentioned earlier in the objects cache discussion as examples of map elements that would be better off imported rather than manually plotted into the scene.
A third way is to use shape files to construct lake objects in TG2. For most projects I guess that manually masking the position of the lakes and adjusting the water level of each lake  until it fits would be OK.
A possible fourth way could be to use shape files to overlay cartographic objects in the model. A semi transparent surface could hower above the landscape or in between objects to illustrate something, a polyline could be seen as a semi transparent wall standing on the surface.   

What do you feel is the basic level of functionality you'd need to effectively work with Shape files in TG2?
As RArcher points out, I too think all editing and preparation of data is better done in a GIS app. There is a lot to implement to achieve this ArcGIS being the hallmark example of how complex it can get. The most important is that the masks made from polygon shapes are properly constructed for use in TG2 and that the projection is such that all layers and object populations imported from shape files stay in alignment with one another, preferably regardless of their extent.

How important is it for you to be able access attributes in TG2 itself, or would you be happy with setting up data for import in a more specialised GIS app?
As I see this, attributes are useful Access to the field names would be necessary for mapping attributes with size, rotation etc.(?) to TG2 equivalents on an object population import.

The same question as above but for operations like buffering or performing queries?
It would of course be neat if several masks could be made in the same operation through selecting by attributes from the same polygon shape file. But then again, it may be that a land use class definition is stored across several attributes so that pre-processing the data is necessary anyway. In many GIS apps you can split the datasets in the same process as the selection.
Buffers are probably handled better in GIS too. Buffering inside TG2 is an intriguing idea though. It is perhaps possible to buffer a mask using an existing function in TG2?
 
The same question as above but with regard to managing projections i.e. would you expect TG2 to work with all projections thrown at it or would you be happy to set up a projection in a more specialized app for importing as something TG2 understands?
Lat/long is fine. UTM too of course. Projecting to lat/long and UTM should not be a problem in most GIS. As for throwing things, it could be that as long as all layers have the same coordinate system and projection they would show up in the same place in TG2.

Erik

[Icegrip]

Im very new to working with Terragen, having worked with forest visualization in VNS which is integrated with shape files. But I think Terragen is way better for creating realistic looking images. I would like to be able to use shape files for being able to better create real life data. The only thing I have been missing with Terragen is the connection with shape files.

I dont know who the basic user of Terragen is now, but I believe that more people could get intrested of using this program if you could connect to real life in an easier way. For example,  Landscape planners, foresters, engineers etc. could use it.

[jlangdon]

I am a long time professional GIS user who has desperately wanted to start using Terragen to create 3D visualizations of geospatial data.  I have some experience with VNS, but the rendering capabilities of Terragen are light years ahead of VNS.  On the other hand, VNS is pretty well integrated with shapefiles and georeferenced orthoimagery.  Frankly, Terragen's current lack of support for GIS data has been what's kept me from diving into Terragen further.  I think if you could really develop the ability to utilize shapefiles in TG2, you would open up TG2 to a whole new market of professional GIS/CAD users.  Speaking as a 10 year GIS veteran, there are unfortunately not too many good choices for rendering GIS data in a high-quality realistic 3D format.  ESRI has it's own 3D rendering app called ArcScene/ArcGlobe (which is absolutely terrible), there's Google Earth and SketchUp for basic low-res stuff, there are the AutoDesk products (also very expensive), and VNS.  VNS is decent, but WAY overpriced, and it definitely has sub-par rendering capabilities.  I would agree with some of the comments on here that you really don't want to go down the rabbit hole of developing too much GIS functionality in Terragen... there are already other GIS software apps that have data manipulation capablities.  ANd to echo RArcher, support georeferenced orthoimagery is really an absolute must.  I could even live without shapefiles for a while if TG2 could import georeferenced raster data.  Here are my answers to your questions:

Are you interested in using Shape files in TG2?
-- Yes, absolutely!

Are you already using Shape files but importing them some other way (as rasters for example)?
-- Not currently using Terragen, due to lack of vector support...

What are you interested in using Shape files for?
-- For point locations of objects, masks for populations, road placement, etc

What do you feel is the basic level of functionality you'd need to effectively work with Shape files in TG2?
-- Probably don't enough about Terragen to use the proper terminology, but I would like to at least be able to use the spatial boundaries of vector objects (polygons, lines, etc) to determine 3D object placement.

How important is it for you to be able access attributes in TG2 itself, or would you be happy with setting up data for import in a more specialised GIS app?
--I would prefer to set up the data in a GIS application (such as ArcGIS or Quantum GIS) and import to Terragen. However, accessing tabular attributes for the purpose of determing object density, orientation, etc, is pretty important.

The same question as above but for operations like buffering or performing queries?
-- Again, I think this can be better handled in a dedicated GIS application.  VNS tries to incorporate GIS functionality like buffering, attributes, etc., but it's definitely not VNS' strong suit.

The same question as above but with regard to managing projections i.e. would you expect TG2 to work with all projections thrown at it or would you be happy to set up a projection in a more specialised app for importing as something TG2 understands?
-- I think if TG2 was set up to handle a small handful of common projections (lat long, UTM, maybe state coordinate systems), that would suffice.

Thanks so much!
Jesse

[Hetzen]

I've been having a serious look at Citygen and an online tutorial/dem has been set up for this Thursday. Anyway, today was the first time I've used OMS data. Wow, this stuff is powerfull.

There's way too much data that I'll never use, but having access to road maps, land usage etc will take TG out of the realms of untouched landscapes and more into keeping of what we see everyday.

I can't really answer the specific questions you've asked Jo, but I'd like to add that having access to vector shapes/masks would have obvious resolution/memory advantages.

A quick question. Which app should I look at to manipulate .OMS data to cut out un-needed data? Is there a good site to help me understand the format better?

Cheers

Jon

[monks]

 Hi, not been around for sometime but I do still fly by to see what's up. Even though I don't publish any of my pics, I do occasionally dabble with TG. :-D I liecensed Deep with Anim when it first arrived.

I'm involved in aproject- have been for the last 6 years or so, to build a world (terrain, map data, etc). One of the things I need is to be able to import coverage directly from the databse: object placemet such as forests, vegatation types, but also shader distribution for wetlands, bedrock types, etc, etc.

For me, I'm looking at doing high altitude renders. So, my problem  is that I need distribution based on geography (latitiude, longitude, geology)- and that requires image maps - I can't recall any function in TG that would currently provide that. I'm having to use imagemaps at the moment and I think I'm going to need quite a few.

I use Global Mapper (which supports a lot of projections) so for me, it's not important for TG to be able to handle lots. The more common ones, such as UTM.

Having worked with very large terrains and vector datasets, working in Global Mapper is always pleasant because it handle them so well. It would be nice to have vector support in TG too. Any scale independence I'm glad to have.

monks

[rcallicotte]

Hate to resurrect old threads, but Shape files would be interesting to me, especially with just finding out about these this week.   

Are you interested in using Shape files in TG2?


Regards,

Jo