Yes I know its a bit wierd to see the word ‘OSX’ in an article by me…I recently bought a Mac for testing InaSAFE and fooling around with FOSSGIS software on OSX. Besides the 2 or three days it took to get it set up as a development machine for FOSSGIS stuff (as compared to... Read more »

This is an update to my previous post “WFS to PostGIS in 3 Steps”. Thanks to Even Rouault’s comments and improvements to GDAL, it is now possible to load Latin1-encoded WFS (like the one by data.wien.gv.at) into PostGIS in just one simple step.

To use the following instructions, you’ll have to get the latest GDAL (release-1600-gdal-mapserver.zip)

You only need to run SDKShell.bat to set up the environment and ogr2ogr is ready for action:

C:\Users\Anita>cd C:\release-1600-gdal-mapserver
C:\release-1600-gdal-mapserver>SDKShell.bat
C:\release-1600-gdal-mapserver>ogr2ogr -overwrite -f PostgreSQL PG:"user=myuser password=mypassword dbname=wien_ogd" "WFS:http://data.wien.gv.at/daten/geoserver/ows?service=WFS&request=GetFeature&version=1.1.0&typeName=ogdwien:BEZIRKSGRENZEOGD&srsName=EPSG:4326"

Thanks everyone for your comments and help!

One of the most popular posts on my blog ever is my article about creating GDAL hillshades: ‘A workflow for creating beautiful relief shaded DEMS using GDAL‘. In the techniques outlined in the aforementioned article, colours are assigned across a continuum to and superimposed on top of a hillshaded model. The GDAL tools in QGIS [...]

I tell you. It always amazes me how much cool stuff you can do with great open source GIS software these days. GDAL is one of those great open source projects that I have just found a great use for (apart from just opening every raster type under the sun in QGIS).

GDAL has the ability to generate contours from a DEM, something that I have always wanted to try for my town but have never been able due to lack of a good DEMs.

Recently we purchased a set of DEMs that cover a large area as part of a study. Each DEM uses a grid size of 1mx1m. Prefect for generating contours.

Using the GdalTools QGIS plugin.

First make sure that you have the latest version of the GdalTools plugin installed (GdalTools should be installed by default with QGIS. If it’s not, search “Gdal” in the plugin installer). Enable the plugin once it’s installed.

Load the DEM into QGIS using the Load Raster icon.

DEM loaded in QGIS

Now head up to the menu Raster->Extraction->Contour

Raster menu in QGIS

Select the settings that you need. For this DEM I am going to generate 250mm contours.

Contour dialog.

Take note of the text area at the bottom of the dialog as that is the exact command sent to GDAL in order to generate the contours. If you take a copy of that you can run it on the command line for batch processing later.

Hit ok.

250mm contours from the DEM

BAM! :)

Using the command line/shell.

Using QGIS for a one off DEM is fine and dandy but what if you have 3000 DEMs that you need to process. To hell with doing that by hand!

Remember the contour tool in QGIS told us the exact command line args to use, so creating a shell script for automating the process is pretty easy.

for f in *.asc
do
  echo "Processing $f"
 gdal_contour -a ELEV -i 0.25 $f $f-250mm.shp
done

The above code will loop though the current directory and process all the DEMs generating 250mm contours for each one. It saves each new contour file as {filename}-250mm.shp. You will need to change *.asc to whatever format your DEM is in.

Copy the above code into a file somewhere and call it generate_contours.sh. This can then be called from the command line using

sh generate_contours.sh

Running sh on Windows

If you’re a windows user you will need to run OsGeo4W Shell in order to use sh.

Loading OsGeo4w shell.

Once the shell is loaded you can just call:

sh generate_contours.sh

Output from running generate_contours.sh

Final remarks

I ran the above sh script on a folder with about 2500 DEMs and it took around 4 hours to complete the whole folder. Of course performance will vary but it seems pretty quick considering.

Once again the possibly to use open source GIS tools to get my work done is amazing.  No expensive software here.

So far I’m not aware of any line smoothing/generalizing abilities using GDAL/OGR.  Although you can import the contours into GRASS and use that: http://grass.osgeo.org/wiki/V.generalize_tutorial

You can also generate the contours using GDAL via Python but that is a topic for another day.

There is quite an old ticket in the QGIS bug tracker relating to the need to be able to select which GDAL drivers to use. The issues is this: GDAL in some cases provides multiple drivers for the same image type. For example, JPEG2000 datasets can be opened using ECW's proprietary driver, KAKADU (gotta love that name!), OpenJPEG etc. The problem is that there isn't actually any way to choose which driver GDAL should use for a given image format - it works on a first come, first served bases. This is a bit of a problem sometimes when you actually want to be using a different driver. In some work I did recently for SANSA (the South African Space Agency), I built a custom version of QGIS for them. They are particularly keen on the JPEG2000 format since it is an open standard and provides good compression. However the custom build of QGIS I made for them includes the ECW JP2000 driver which unfortunately chokes and dies (no fault of GDAL, it happens down in the ECW code itself) when trying to open their JP2 imagery. Also we would prefer to use a FOSS driver for JP2 in keeping with the general ethos of being a FOSS project. So all this prompted me to implement the needed support to let you disable GDAL drivers at runtime in QGIS which is illustrated a little by the screenshots below.

By default GDAL want to use the ECW JPG2000 driver...(click to enlarge)

Using the options dialog, you can now disable a driver....(click to enlarge)

After the driver is disabled (and restarting QGIS), the alternate JP2 driver is available (click to enlarge)

Today I committed changes to QGIS master that I started working on at the Lisbon hackfest and finalised this week (along with a unit test yay!). The changes implement improvements to the way in which statistics are gathered for rasters. First a little history: the raster stats code in QGIS is some of the earliest work I have done in QGIS. The code was also hugely inefficient - basically it completely scans every pixel in the file twice! It does this because it extracts minima, maxima, stddev etc. from each band in the raster - and the algorithm used to calculate the stddev requires first calculating the sum of means, hence the second pass. Of course we use the underlying GDAL library to traverse the raster, but we completly bypassed the GDAL stats gathering functions. Now from my testing (see below) the difference of gathering stats in QGIS as opposed to directly in GDAL was not that great. However, the real killer is the fact that QGIS had no reasonable implementation to cache these statistics - either within a session or between sessions. This resulted in quite a lot of  complaints about the slow loading of rasters in QGIS and the spontaneous recollection of stats when making various changes to raster symbology etc.

With the release of 1.7, Radim Blazek implemented 'proper' support for raster providers so that they follow a similar model to the vector providers in QGIS. This opens up room for provider specific optitmisations and for implementing new providers in the future e.g. for PostGIS Raster, Terralib etc. At the Lisbon hackfest, I refactored the provider interface so that it provides a default implementation of stats gathering which can be overloaded by specific providers, each then being able to use best practice for that datasource when gathering stats. I then updated the GDAL Provider to use gdal native calls to fetch statistics.

When GDAl gathers stats, it stores them in a file named after the source raster but with a .aux.xml extension. As a bonus, GDAL also stores histogram data in this file. Thanks to some excellent tips from Etienne (Sky?) on the QGIS mailing list my implementation was further improved so that it actually uses these cached stats from GDAL properly. So let's see how performance is improved with these changes. I tested using a ~650mb tif hillshaded DEM from one of my clients.

 Added bonus: My changes also use a more efficient approach for obtaining min/max for a band (before all stats are gathered) and also histgram calculations are cached now. So after the initial gathering of stats and histograms, all activities when working with GDAL rasters should be near instant now. By the way the 1 in the graph above is rounded up - in my usage it was near instant in most cases.

I need to still evaluate whether it will be good to backport these changes to the 1.7 stable tree, so for now if you want to benefit from these changes, you need to be using a nightly build or build from source.

Every 6 months we hold a hackfest / QGIS meeting and many of our users generously donate towards it. I hope the above article gives those who do support our meetings a sense of the tangible benefits that arise from these meetings - there have been many other improvements coming into master that arose from the last hackfest that we all get to enjoy! If you would like to help towards the costs of the next hackfest, visit http://qgis.org and click the 'Donate!' button on the left hand side. Have fun!

Another cool open source project that I have become a part of (as a QGIS packager and tester) is the OSGeo-Live project.  The OSGeo-Live project is a live DVD/USB/Virtual Machine built on xUbuntu(striped down Ubuntu linux) that has a lot of cool open source geo spatial programs all set up and ready to use.

The OSGeo-Live project contains:

• Browser clients
• A small sample of crisis management software
• All the popular database engines (PostGIS, SpaitalLite etc)
• Pretty much all the open source desktop GIS apps (QGIS, uDig etc)
• Open Source GPS navigation apps and globes.
• A collection of handy spatial tools
• A ready to go web services ready to try in your browser or desktop GIS.
• Some sample data to get started with for each project
• And quick starts for each program.

The full list of software contained on the OSGeo-Live project can be found at http://live.osgeo.org/en/overview/overview.html

This is a good project if you want to get into the OSGeo tools and experiment but don’t want to install them on main machine until you know what you need.

As it is a live DVD/USB/Virtual Machine some apps will run slower than what they do on a native install but overall the speed is usable and good enough for testing.

Even better is that it was born in Australia :)

The project is also commercial supported by a Australian company http://lisasoft.com

So give it a try if you are interested in the OSGeo movement, which you should be if you are reading my blog :)

If you follow the osgeo planet blog aggregator, you may have noticed Frank Warmerdam's recent blog post mentioning that he is off to work for Google. If I think back to the blog posts I have written, GDAL features in a great many of them - its a mainstay tool for me and barely a day goes buy that I don't gdal_translate some file or what-have-you. Hell, I just finished running a batch job that ran for about 3 weeks comprised of nothing more than bash script and GDAL commands.

In QGIS too, GDAL is one of the key underpinnings of our project - it is only now 8 years or so into the project that we are starting to abstract the use of GDAL for reading rasters away a little to make space of other raster provider types. And of course OGR is one of the main ways that our users interact with vector data from within QGIS.

Personally, Frank has also been extremely helpful whenever I have wondered into the #gdal IRC channel looking for help. Reading his blog post made me think that Google doesn't quite realise yet just how lucky they are to get him. If Google has any heart, they will make Frank a 'minister without portfolio' and just tell him to keep working on GDAL and enjoy the free food in their cafetarias. Here's hoping at least that he still gets to spend a little time on the software that so many of use love using and depend on.

Note:The following post requires gdal 1.8

A lot of times we need to send/use 3D dxf files, we used to use FME however FME is not free or cheap. So I went looking for a free solution.

If you have gdal 1.8 it’s just one simple command line run using, what is becoming my favorite GIS tool, ogr2ogr.

All you have to do is run:

ogr2ogr -f "DXF" {outFile} {inFile} -zfield {ColumnWithZValue}

So in my case I ran:

ogr2ogr -f "DXF" C:\Temp\contourswarwick.dxf C:\Temp\Contours.TAB -zfield Height

I haven’t fully tested it but {outfile} can be any file ogr supports.

and the output:

Top view of contour layer

The side view of the above image.

In the words of, the not so great, Charlie Sheen. WINNING!

Happy mapping :D

Suppose you have a DBF (.dbf) file containing X and Y values that you want to import and save as a spatial layer.

QGIS doesn’t support direct import of a DBF file as a map layer, however, we can use some command line magic to convert it to a CSV file and then use the Delimited Text plugin to get the job done.

Your DBF file should have an id for each record and fields containing X and Y values. If it has additional fields that should be OK as well.

First convert the DBF to a comma delimited file using ogr2ogr:
ogr2ogr -f CSV my_csv my_data.dbf

If you don’t have ogr2ogr see http://www.gdal.org/index.html.

This will create my_data.csv in the my_csv directory. You are now ready to bring it into QGIS.

Here are the steps to import the CSV:

1. Start QGIS
2. If not already enabled, use the plugin manager to enable the Delimited Text plugin
3. Click on the Delimited Text icon in the Plugin toolbar or choose it from the Plugins menu
4. Browse to the location of your CSV file
5. Enter a name for the layer
6. Under Selected delimiters, check Comma
7. If your X and Y fields aren’t automatically determined, set them using the drop-down boxes
8. The sample text should show how the file is being parsed—if it looks right click OK, otherwise adjust the settings
9. The layer is added to QGIS

At this point the layer behaves pretty much like any other QGIS layer. To save it as a shapefile, right click on its name in the legend and choose Save as…

The 1.8.0 release is a major new feature release with the following highlights:

• New GDAL drivers: GTX, HF2, JPEGLS, JP2OpenJPEG, JPIPKAK, KMLSUPEROVERLAY, LOS/LAS, MG4Lidar, NTv2, OZI, PDF, RASDAMAN, XYZ
• New OGR drivers: AeronavFAA, ArcObjects , GPSBabel, HTF, LIBKML, MSSQLSpatial, NAS, OpenAir, PDS, PGDump, SOSI, SUA, WFS
• Significantly improved OGR drivers: DXF, GML
• New implemented RFCs
• New utility: gdallocationinfo

More complete information on the new features and fixes can be found in the GDAL/OGR 1.8.0 Release News.

<p><p><p><iframe src="http://rcm.amazon.com/e/cm?t=songsforkids-20&o=1&p=8&l=bpl&asins=3642094252&fc1=000000&IS2=1&lt1=_blank&m=amazon&lc1=0000FF&bc1=000000&bg1=FFFFFF&f=ifr" style="padding-top: 5px; width: 131px; height: 245px; padding-right: 10px;" marginwidth="0" marginheight="0" align="left" frameborder="0" scrolling="no"></p></p></p>