Planetside Software Forums

Introducing TerraRoad - Terragen roads where you want them!

https://github.com/abe-mart/TerraRoad

I got tired of not being able to control road placement in Terragen, so I cooked up a Python script to help.  TerraRoad combines a heightfield exported from Terragen with an SVG vector path, and produces a new heightfield with the road surface smoothed.  It also exports masks that can be used for shading the road, border, and markings, as well as cut and fill areas.  

The link above has the Python source code, and install instructions.  I believe this should be cross-platform, but I've only tested it on Windows.  

There are some limitations.  This is heightfield and image based, so you're limited in area and resolution unless you want gigantic files.  This is particularly noticeable for closeups of the road lines.  They look decent from far away though.  The script assumes a single road for now, and no crossroads.  

This is early in development, so feedback is welcome and needed.  Don't expect a finished product, bugs are expected at this point!  A good first test would be to see if I've missed any important steps in the install instructions linked above.  I'm open to ideas on how this could be improved as well.  

   

This looks interesting, and probably very useful. I feel like this is the sort of modern functionality Terragen should have in it's package. These little features will add up to really comprehensive world building.

I do have one request: Allow PNG/BMP/TIFF support for images. Though, if I know Python right, that may require then converting the formats to SVG anyway. I know I have SVG support in CS5 but not my older Photoshop.

PS; May I add this resource to the Terragen resource listing on NWDA?

Sure, feel free to add to your list, though like I said before, active development is in progress, so no guarantee of functionality at this point.

I'm using the Imageio library for image import and export, so any of these formats (https://imageio.readthedocs.io/en/stable/formats.html#single-images) should be possible.  The road shape is a little trickier.  It's important that it's in a vector format rather than a bitmap, because I'm using the svgpathtools (https://pypi.org/project/svgpathtools/) library to load in the road path and calculate the normals and offsets that define the road and shoulder.  My intention is that people would use the pen tool in Photoshop or similar to draw a spline over the terrain where the road should go.  

On a side-note if you haven't checked out Affinity Photo yet I'd highly recommend it.  I switched over a while back and have rarely missed Photoshop at all, which is awesome considering the price.  I've been using it to make my .svgs.

Good that you take it off the alpha part. You'll get a lot more feedback I guess. I still didn't have time to try it, sorry.

I tried to install it, but I got a few warnings and error with the prerequisites which I don't understand, though It's late here.

PS I do have Affinity Photo, but always forget. Lol I also, am embarrassed to say, I still have yet to fully grasp the pen tool. Lol Like people always use the pen tool to cut out things. I painstakingly use the polygon lasso tool with the pixel ruler. Lol

Quote from: pip outputWARNING: The script f2py.exe is installed in 'C:\Users\WAS\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts lsm2bin.exe, tiff2fsspec.exe, tiffcomment.exe and tifffile.exe are installed in 'C:\Users\WAS\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts imageio_download_bin.exe and imageio_remove_bin.exe are installed in 'C:\Users\WAS\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
ERROR: Could not install packages due to an OSError: [Errno 2] No such file or directory: 'C:\\Users\\WAS\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\site-packages\\sklearn\\datasets\\tests\\data\\openml\\292\\api-v1-json-data-list-data_name-australian-limit-2-data_version-1-status-deactivated.json.gz'

WARNING: You are using pip version 21.1.3; however, version 21.2.4 is available.
You should consider upgrading via the 'C:\Users\WAS\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\python.exe -m pip install --upgrade pip' command.

Thanks for testing.  The warnings aren't a problem, it should still run fine, and you can suppress them with the suggestions it gives if desired.

The scikit-learn error is a problem.  It looks like the package hits Window's default 260 character path limit.  https://github.com/scikit-learn/scikit-learn/issues/19306.  You can remove the limit in Windows 10: https://www.howtogeek.com/266621/how-to-make-windows-10-accept-file-paths-over-260-characters/.  That's an annoying extra step for everyone to do though, so I'll probably try to remove the dependence on scikit-learn.  I think I only use it once in the code, because I was too lazy to write my own function to scale numbers between 0-1 Lol.  Should be pretty easy to swap out.

I haven't used the pen much before this either, but the pen skill level required for road drawing isn't too bad.  I found in Affinity at least you can select the pen, then click the little 'Smart Mode' button towards the top, then click a few times.  It auto creates a curve through your points, and you can hold Ctrl to edit.  Then save as svg, and the script uses your curve as the center of the road.  I'll try to make a little getting started video at some point here.

UPDATE:
Just pushed a new version of the code with the scikit-learn dependency removed.

Windows 10, python 3.9 installed. Won't start, no error messages.

Thanks for testing mhaze.  No error message, and no pop-up window?  And that's from running 'python TerraRoad.py'?  Strange.  I've tested it successfully on a couple of computers, but I'll try it on my wife's laptop later to increase my chances of finding the problem.  I'll try to put up a video tonight as well to show the expected behavior.  

We'll get this ironed out eventually.

I got the latest version to work, though I needed to install 3 of the dependencies that already were installed. Lol No clue.


To run the software (from my desktop) I just did the following (first step is unnecessary for pretty much any other Windows OS; I have CMD customized):
C:
cd Users\WAS\Desktop\TerraRoad-main
python TerraRoad.py
which through the traceback let me know any problems I needed to fix.

---

Ok. I guess I need more help... do you have a tutorial for creating paths that are equal in width? I have no clue what I'm doing, and the pen tool just acts as a lasso tool when you go to fill it or mask with it.

---

Quote from: WAS on August 23, 2021, 02:21:53 PMI got the latest version to work, though I needed to install 3 of the dependencies that already were installed. Lol No clue.


To run the software (from my desktop) I just did the following (first step is unnecessary for pretty much any other Windows OS; I have CMD customized):

Code Select Expand

C:
cd Users\WAS\Desktop\TerraRoad-main
python TerraRoad.py
which through the traceback let me know any problems I needed to fix.

And we have a winner!   ;D  First successful launch is a good first step!

Quote from: WAS on August 23, 2021, 02:59:06 PMOk. I guess I need more help... do you have a tutorial for creating paths that are equal in width? I have no clue what I'm doing, and the pen tool just acts as a lasso tool when you go to fill it or mask with it.

So actually all you need is the line in your first picture.  The .svg defines where the road should go, and then TerraRoad basically applies a 'stroke' to it to give it width.  You should be able to just take what you had in your first picture and export it as an svg.  I've been disabling the background first, but I don't know if that's necessary, I'll have to test. 

I think it will work with the road extending outside the terrain boundary, but I haven't tested that yet, so if it gives trouble try bringing the whole path inside the borders.

Thanks for the quick response! All making sense to me now. Will hopefully return with an example image.

PS I added this tool to the resources' page on NWDA under Terragen: https://nwdagroup.com/resources/

WAS tried your method but was told that python is not recognised as a command!

I recorded a quick video walkthrough that I'll try to upload today.  

In the meantime try running just 'python' from the command line.  If it doesn't print out something about Python, your Python install either isn't set up right, or is set up differently than we're expecting.  

WAS and I have been working through some separate issues trying to export .svg files from Photoshop.  It seems that Photoshop just saves a .png and calls it a .svg, which doesn't work because it isn't actually a vector.  I've had good luck in Affinity so far, but I'll explore some alternatives for creating the spline path as well.

Interseting - python is recognised by the system. If it's any help when I run the program a box flashes up and immediately disapears.

Quote from: mhaze on August 24, 2021, 03:06:51 PMInterseting - python is recognised by the system. If it's any help when I run the program a box flashes up and immediately disapears.

What output do you get when you run the pip install instructions?

That is helpful actually.  It could mean that the script is hitting an error, and then immediately closing before you get a chance to see what it was.  I might be able to make a debug version that will hold the window open if it hits an error.  Was this from running "python TerraRoad.py" from the command line, or from clicking the file?  (Not sure why the forum really wants that filename to be a link, sorry)

Here's a rough video showing my workflow for installing, creating the input files, and running.  https://youtu.be/tF4_IzExQv8

Both!

OK! problem solved. Thank for the help and YouTube video, it helped no end, I was not loading the dependencies correctly. Now to work out how to use the program!

Quote from: mhaze on August 25, 2021, 01:26:37 PMOK! problem solved. Thank for the help and YouTube video, it helped no end, I was not loading the dependencies correctly. Now to work out how to use the program!

That's what I figured.


I had issues with the prerequisites too. Strangely, installing the prerequisites on their own one by one produced no errors.

Quote from: mhaze on August 25, 2021, 01:26:37 PMOK! problem solved. Thank for the help and YouTube video, it helped no end, I was not loading the dependencies correctly. Now to work out how to use the program!

Glad the video helped!  Take it for a spin and let me know what works and what doesn't. 

WAS already pointed out that some of the masks are blurred too much.  I usually run them through a color adjust in Terragen to tighten the edges, but I'm thinking I will also add another tab that lets you choose which masks you want, at what resolution, file format, and how much blurring/smoothing to apply to each.  The mask blurring is to try to help with aliasing, but it seems that the road mask needs a different amount of blur than the lines for example.  At some point, shading with image masks may just be too resolution limited, but I can't think of a good alternative.

I'm also planning to add some images to the docs to explain better what the settings do.

Quote from: aknight0 on August 25, 2021, 05:31:28 PMAt some point, shading with image masks may just be too resolution limited, but I can't think of a good alternative.

I wish we had velocity type warping. I imagine TerraRoad could output a warp map, which could be used to warp things in Terragen to the road (from 0,0,0).

The biggest issues with roads in TG currently is you can't plot them, additionally, the warping in TG is not very good, you got one flavour, and it doesn't warp road proportionately.

Here's a couple more samples on a Gaea terrain.  The sideline masks also work pretty well for dirt roads if you make them wide and blurry.  It's surprisingly difficult to get natural looking angles for a road climbing a hill.  I'll have to look at some more pictures, or find a civil engineering book or something.   :D

I'm working on an update that allows you to set the resolution and blurring for each mask individually, as well as adding 16 bit PNG output for the masks, which seems to help somewhat with the road lines.  

 

This is a very promising and probably really helpful plugin for TG. Even for paths Id imagine too. Still need to tinker some more.

That is a great example of its possibilities! Have you talked to Matt about ways to integrate this into TG directly?

Great work!

Confirmed that it could be launched on my macbook.
(used python3 command to launch)

I'll try to generate textures.

Just uploaded version 0.2 to the GitHub page.  Here's the changelog:

### Added
- Save Terrain as JPEG button for easier road drawing reference.  (Saves normalized grayscale image to selected output folder)
- Mask Settings tab with individual blur and resolution settings for each mask.
- Check at startup to see if an small EXR file can be read.  Prompt user to allow automatic install of FreeImage EXR library if needed.

### Changed
- Rearranged some of the basic and advanced settings based on the ones I use more often.


There's a bottleneck right now with blurring masks upscaled to large resolutions.  Apparently doing a gaussian blur on a 50k image uses a huge amount of memory, goes into swap, and is very slow.  I recommend running with low res masks until you get the heightfield right, and then turn the mask resolution up.  I also definitely recommend using a color adjust after the image map to fine tune the masks.  

Dune, I've talked back and forth with Cris a few times about this, but not to Matt directly.  I think he's been on vacation.

Here's another quick sketch.  Also, I started working on some better documentation here: https://aknight0.gitbook.io/terraroad/

That's definitely a very good example project right there. Thats cool, and literally something I have wanted to do in TG.

Yeah, that's an excellent use of your road app! wanted to do that years ago but couldn't work it out! Most of the problems I've encountwerd with the app were due to my own stupidity! but helped my sort it out. The .exe flie from heightfield generate has to be carefully setup otherwise nothing else works! Used Affinity for the .svg file as I could not work out how to export it from Photoshop. Oh, later displacements messed up the white lines hence the request for a larger road mask - then I can rough up the verges without messing up the white lines. I could probably produce something in Photoshop but it would speed up the process it I had an extra mask.
It would be useful facility if I could up date just one parameter instead of waiting for 4/4 mins to up date them all (8191x8191 HF terrrain)

A couple ideas on the verge.  One would be to put a color adjust on the road_and_shoulder_fade mask.  That mask does a gradient from the center line out to the edge of the shoulder.   With the right black and white points you should be able to create a mask a little larger than the road.  Another idea is to re-run the whole thing in a different output folder with a larger road width, and use that larger mask with your original terrain.  I'll add some pictures of the different masks to the docs, and we can add more masks if needed.  

The latest code has options for choosing which masks to export, which should save some time.  Since you have the prerequisites installed, you should just be able to re-download the new code and run it.  The heightfield is still re-processed every time, maybe I should make a separate Create Masks button.

I haven't been able to figure out how to export a good .svg out of Photoshop either.  It's strange, usually my problem is that Affinity is lacking a feature from Photoshop, not the other way around.

I realized I could use a Colour adjust about 15mins ago playing with it now.

I tried to install it, but I got a few warnings and error with the prerequisites which I don't understand, though It's late here.

I just added a new verge mask which can extend out beyond the shoulder if desired.  

jaattvhishal, the warnings should be fine.  Can you post the error you're getting?  Also check out the video I posted earlier in this thread for a rough walkthrough of the installation.

Great,that will be useful.

Great project. The mountain road needs to extend gently along a certain height. I used a contour node paper as a reference, so that the layout of the mountain road can be more reasonable.

---

That's a good idea. Though, I'm not sure, but I think you'd want your roads to hug between the contour lines. As the area with no lines indicate the "flat" areas of the terrain where the contours indicate levels of altitude sloping? Maybe I'm backwards there.

I actually have adding reference contours on my to-do list, since I've also found that it's difficult to eyeball a consistent elevation on a mountain.  Should help with creating more natural looking paths.  It will probably be a second button next to the 'Terrain to JPEG' button.  I do like the shadows on your top-down render though, that gives some nice context.

WAS, usually the contours are lines of constant elevation, spaced evenly in elevation, so lots of lines close together means a steep slope.

Quote from: aknight0 on September 01, 2021, 03:05:32 PMWAS, usually the contours are lines of constant elevation, spaced evenly in elevation, so lots of lines close together means a steep slope.

Oh yeah, I know that, but what I mean is, like you said, they're a constant denotion of elevation frequency, so the largest intervals, and the centre spacing between them should be the flattest areas of the topology. That's how I've always read hunting maps, and was pretty consistent with the real terrain.

Oh, I see what you mean.

do you get when you run the pip install instructions?

when you run the pip install instructions?

Quote from: Toggle on November 08, 2021, 11:04:03 AMwhen you run the pip install instructions?

Hey Toggle, are you using a translator? It's a hard to understand your posts.

Quote from: WAS on November 09, 2021, 03:33:07 AMHey Toggle, are you using a translator? It's a hard to understand your posts.

I think, it's spam. Lots of strange comments.

Spammer for sure. Nonsense sentences, nothing useful anyway.

I suspected maybe something like that, but spammers don't often set avatars, and usually quicker about editing in links.

They apparently evolve. Before no time they have something real to say, like a fullgrown Terragenist, and then edit with a link :'(