Yet Another GIS Blog
GIS, Geography, Programming, and Neogeography

Why every GISer should learn Python

Monday, 8 March 2010 12:16 by boxshapedwo

What the hell is Python?  You may or may not be asking.  Well, it is a "high-level programming language" according to wikipedia.  Not so long ago, to me it was just some other thing that ESRI was installing on my machine in an already excruciatingly long process.  Then I actually tried to use it for some geoprocessing in ArcToolbox, and actually have found it quite a valuable skill to have.  I do find the syntax to be cumbersome to use, but my programming started with Visual Basic, so it is much more concise than VB.  Anyway, here are my reasons.

  1.  ESRI ArcGIS is using it for a lot of their ArcTools.  They seem to be supporting it at least for the next few versions, unlike VBA. (Keep reading as I discuss nonESRI reasons further down)
  2. You can do a lot with Python outside of the ESRI environment too.
  3. You don't need ArcGIS in order to do geoprocessing with it.
  4. You get to use cool libraries like matplotlib to create your charts instead of Excel.
  5. It is used in a lot of web programming, like MapServer.

Why am I recommending getting started with Python, when I invested time and effort in creating VBA tutorials (as yet unfinished).  VBA is still useful as the syntax is similar to VB.NET.  But VB.NET has a bigger learning curve.  Python will be supported for a while, and you can get started with it while staying within the ArcGIS environment.

How should I get started then?  Assuming you are using ArcGIS 9.3.  You should install PythonWin as well as that came on the installation disc separately.  If you don't have access to that disc, or admin privledges, I'll describe this using something else.  Pick a task that you use ArcToolbox for often.  Let's use Clipping as an example that should be available to anyone with any license.  Add two generic polygon layers to work with.  Add ArcToolbox if you don't have it open already, right click and create new toolbox.  Right click on your new toolbox and add a new model. Drag the Analysis Tools -> Extract -> clip tool to the model and you should see a square and oval.  Double click on Clip and fill out the form as you normally would.  Click ok and you should see two blue ovals added.  Go to Model --> Export --> To Script --> Python.  Call it what you want.  Discard the model.  Go to where you saved the file with a .py extension.  Right-click on it and select edit with IDLE.  Two windows open.  A note here.  Version 9.3 of Arc uses version 2.5 ofPython, and only 2.5.  Version9.2 uses only 2.4.  You can't use a different version of Python with a different version of Arc.  Python will install multiple versions rather than overwrite existing ones.  It is possible that you still have 2.4 from arcgis 9.2 still installed.  Or another program might have put 2.6 on.  Look at the window opened that is called Python Shell and it will tell you what version.  If it opened with the wrong version  Go to Start -->  All programs -->  Python 2.5 --> IDLE.  When that opens got to File--> Open and open up the Python file that way.  It is really important that you get the version right, or else your tool will just crash.  Now you have your first bit of code.  I suggest you go through it line by line to understand what is happening:

 # ---------------------------------------------------------------------------
# clip_example.py
# Created on: Tue Mar 09 2010 12:06:20 PM
#   (generated by ArcGIS/ModelBuilder)
# ---------------------------------------------------------------------------

# Import system modules
import sys, string, os, arcgisscripting

# Create the Geoprocessor object
gp = arcgisscripting.create()

# Load required toolboxes...
gp.AddToolbox("C:/Program Files/ArcGIS/ArcToolbox/Toolboxes/Analysis Tools.tbx")


# Local variables...
statesp020_Clip_shp = "C:\\WorkSpace\\FlowMap\\statesp020_Clip.shp"
statesp020 = "statesp020"
statesp020_Project = "statesp020_Project"

# Process: Clip...
gp.Clip_analysis(statesp020, statesp020_Project, statesp020_Clip_shp, "")

The # is a comment and it is not processed by the compiler.  Import loads different libraries.  ESRI has a special one called arcgisscripting that you use to create a geoprocessing object (gp).  This gives you access to all the ArcTools.  Paths to files use either the forward slash / or double backslash \\ but not a single backslash \.  This is because strings use this slash as a special character and it would confuse python if you did this.  If you change the statesp020 and statesp020_Project (your variable names will be different) to equal their actual path to the file, then, in theory you should just be able to go to Run  --> Run Module and it will perform the operation.

Your next task:  Figure out how to loop through all the feature classes in a folder and clip them.  You already have one important line of code gp.Clip_analysis.  You just need to do the looping bit.  Follow the isntructions on this page for more information.  As always, the ESRI help is quite good and extensive (which does make it difficult to find things).

 

What if I don't use ESRI, and I use MapInfo or Manifold you are now asking.  Well I'm sorry for you.  Just kidding, those programs have their merits too, no really they do.  :).  Unfortunately, you cant use python with those programs, but there is something called GDAL that is an open source geoprocessing library (and more) that you can use instead.  Here is a lesson on installing  GDAL for python.  And Chris Garrard has been generous in posting his course presentations that show step by step how to work with python and GDAL.  I recommend even for ESRI users giving that a once over.

I hope this has convinced you and given you a start on using Python with GIS.  There are plenty of tutorials that will give you more of the basics of Python Syntax, but I find it better to start with the GIS stuff, which I'm familiar with, and then look up the Python bits I'm less familiar with.

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Tags:  
Categories:   Programming
Actions:   E-mail | del.icio.us | Permalink | Comments (1) | Comment RSSRSS comment feed

Examples

Tuesday, 23 February 2010 11:42 by boxshapedwo

While, I don't have anything against FortiusOne or the "neogeography" work that they create.  I do have something against shoddy examples.  Here is a shining example from a recent feature they have added to their GeoCommons project.  First, though, I'm not entirely sure what a proportional shapes map is supposed to be.  I don't know if the values are changed.  In their description of map types, it sounds appropriate but the image they show is just a "color map."  So I'll let that go.  The problem I see is the example they are using.  The map the total number of children under 16.  The TOTAL number of children under 16.  Look at what states are the darkest and have the highest number of children under 16.  Wow, you just created a map of the most populous states.  They should have mapped the proportion of 16 and under.  I realize this is just an example.  When you are demonstrating something to novices, you should in the least create correct examples so you are not at fault for showing them incorrect information.  Or, at least, qualify that you are a novice.  Google Visualization API does this same thing in their examples.

Now...Don't read any other entries in this blog or else I will probably be found at fault for this too.  I hope that I qualify that I'm not a programmer at least on my programming tutorials. 

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Categories:   Neogeography
Actions:   E-mail | del.icio.us | Permalink | Comments (2) | Comment RSSRSS comment feed

Mapping Framework

Tuesday, 23 February 2010 08:25 by boxshapedwo
Here is an Actionscript based mapping framework that was recently released.  It too uses the vanrijkom toolset for accessing shapefiles.  No projection control though.  Via FlowingData.

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Copying the NYTimes

Wednesday, 10 February 2010 15:32 by boxshapedwo

A recent Cartotalk post caught my attention about a new Infographic at the NYTimes.  If you don't want to follow the link, the basic gist of it is they have set up a series of interactive 3D maps of the Vancouver Olympic area.  As per usual, they are well made and fast.  The poster's original question was how they were made.  I responded with my thoughts, but I really don't know how it was made.  It got me thinking, and wondering, if I could do something similar.  Well, not one to let silly things such as inexperience or lack of knowledge get in my way, I decided to go for it.  The area I chose to work with was the Huangshan Mountains (Yellow Mountains) in China.  The tools I used, and had at hand were:  FlashDevelop (Flex 3.3 SDK), Papervision (most recent swc), Global Mapper (version 9), and Google Sketchup version 7.  I had a leg up in that I've worked with Papervision when experimenting with Augmented Reality (FLAR).  I don't have much Google Sketchup experience so it was a bit of a crash course.

 To obtain the data, I used Global Mapper.  I highly recommend this program whether you have an expensive GIS or not.  This program will save you loads of time converting between files, downloading files, and clipping them.  It's also really fast.  I just downloaded SRTM and Landsat data for my project area. I ended up picking an area that was quite hilly and exagerated.  This actually became a problem later. 

In order to use the terrain with Flash and Papervision I needed it in the open Collada format.  The only program I had to create collada files was Google Sketchup, and just the free versio.  Luckily there were a couple of formats available for the import: dem, dxf, and 3DS.  Global maper exports to both dem and dxf.  I initially tried a DEM but that created a really ugly model.  Then I played around with DXF mesh and DXF Face.  Both seemed to work better.  Because I had picked this SRTM data in an area with great elevation variability I needed to reduce the elevation height to create a reasonable mesh.  This was done through Global Mapper's control panel and the layer's properties.  I set mine to .05 scale factor.  When exporting to the DXF 3D face file, I also changed it so that it had a 250 metre by 250 metre grid.  The key here is that you export files that are manageable.  I dxf file will probably need to be less than a mb, and the image I used was around 3mb (which seemed ok).

I'm no Sketchup expert, but was able to figure things out by following this tutorial.  The key steps to follow are the smoothing step and edit group before adding the texture.  When you import your dxf file it will not be grouped, so you need to do that step yourself.  While I was in sketchup I used those tools to add a square block underneath.  I used the Intersect with Selection tool to erase the parts of the block I didn't need, and added a different texture.  Once I was happy with the model, I exported it to a Google Earth KMZ file.  For those that don't know, KMZ is just a zip file, so change the kmz to zip and then open this.  Your model should be stored in a models folder with a .dae extension.  Textures are stored in the images folder. From there I just parsed them using papervision.

There are some really good papervision tutorials available here and here.  That's where the PaperBase.as file originated.  Because of some earlier experimenation, I actually found DAE worked better than the Collada parser.  The code is available here.  A demo version is available here.  Click and drag the mouse to cause the camera to move.

I make no claims that this is a replica of the NYTimes piece, but hopefully would get you started.  The model needs to be placed properly and the interaction needs to be worked on.  I was primarily concerned with getting terrain data into flash, from there it is up to the real designers :).  Also, considering I got this up and running in a few hours speaks to the quality.  But the ease of which it was done is a testament to the power of papervision and Google Sketchup rather than any particular skills of mine.

 

Feel free to take the code and use it for whatever, there isn't a licence, but I claim no responsibility and there is no warranty available.  Use at your own risk.

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Flex Fanboy?

Monday, 1 February 2010 16:57 by boxshapedwo
I never really thought of myself as a Fanboy of anything...I'm too cynical perhaps.  But with the unfortunately named IPad arriving, the Flash haters have come out of the woodworks saying that Apple's continued lack of support of Flash will be the end of flash.  Their reasoning?  Something called HTML 5 and the video tag.  The nice thing about about HTML 5 is that when it finally arrives then you will theoretically be able to put a <video> tag in your HTML and the browser will just play it.  Of course, that makes light of the many varied video formats available, and Apple is really only supporting one video codec for its browser.  While I DO think HTML 5 is a great direction head, I think it is almost as mythical as the IPad was two weeks ago ("it will do this and this and that and this").  I find it annoying how quickly everyone is to criticize flash.  Some truly wonderful things have been created with flash; one only needs to search the NYTimes for interactive to see what I mean.  And, well, frankly the HTML 5 Canvas element is just a glorified Flash sprite with a javascript backend.  Is really the only advantage of HTML 5 is that the browsers will support it and you don't have to install an extra plugin?  That is, IF they support it at all, Internet Explorer doesn't yet.  You also hope that each browser will support the same specification, otherwise you are stuck in a potential AJAX hell of writing code for each browser's implementation.  That's what happened with Internet Explorer.  At least with Flash you know it will run the same in each browser.  People point to performance, but I haven't seen anyone do anything with performance and HTML 5, so who knows if it will be better.  Playing video and games is a drain on the battery regardless.  I don't deny that Flash will go away eventually, especially if HTML 5 turns out to be as good as everyone says it will be.  And, transitioning to HTML 5 and javascript won't be too painful given the similarities with Actionscript. I just don't think that people should criticize Flash, because in the end it does give us good web experiences.

Be the first to rate this post

  • Currently 0/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5
Categories:   Neogeography
Actions:   E-mail | del.icio.us | Permalink | Comments (0) | Comment RSSRSS comment feed