Just had a heart-attack but now I'm ok.  I have a map in ArcGIS that uses several joins to link up by statistical boundaries with the data stored in a DBF file.  I realized I was using the wrong score, and needed to update it.  Rather than do all the joins and symbology over, I opened the dbf file in Excel and changed the score field...That is I opened it up in EXCEL 2007 (THE HORROR!!!).  Sure, it opens up fine, but when you try and save to a dbf, it doesn't let, or even have the option anymore.  Curses!   Luckily, the interweb had a solution for me.  Save the file out to an Excel file (xls instead of xlsx just to be on the safe side), open it up in ArcCatalog (can do this in ArcMap as well), then simply export it to a dbf file.  Everything should be the same as before.

First the good news.  I am quite pleased that this blog was referenced by Kelso's Corner, a very enjoyable cartography oriented blog, not once but twice.  Unfortunately, the flow map entry on my blog wasn't particularly interesting.  However, it did get me off my duff and start to look at "porting" the idea to something that works with shapefiles.  I've gotten as far as creating the hierarchical clustering, then I'm a bit lost in how the paths are drawn based on these clusters.  Plus I'm using it as an excuse to study up on c# which has become much easier syntax wise since I've learned a lot about AS3.  I'm no expert in any of the languages by any means.  Also, I might have a little problem in the hierarchical clustering too that arose when I tried it with some more complex data - all 50 US states.  Of course, I've been busy with paid stuff so I can't really focus on this at the moment...now I'll forget where I left off and then have to start over again :).

Now for the real part of this blog.  I don't work with GPS units too often.  Other than downloading data.  Well with the current project I'm working on, I will be doing some ESL GPS training (that's English as a Second Language).  We purchased two Garmin 60CSx units through a US based colleague, and hence have very dodgy Australia data.  So I thought to myself, "No big deal, I'll just use my own data."  Phsssh, adding data to a Garmin GPS is a huge pain in the ass!  Here are the steps I went through to get the data into the Garmin GPS.  I took some of this from site.  To do this required 4 separate pieces of software - Mapsouce (comes with Garmin GPS), Global Mapper (I used version 9, which I have a license for), cGPSMapper (free edition), and finally MapSetToolKit.

Step 1:  Make sure everything is installed on your computer.

Step 2:  Open Global Mapper, load your data.  I used shapefiles, and changed the projection to WGS84.  To change the projection, you use the configuration dialog.  I'm presuming some familiarity with Global Mapper. 

Step 3:  Make sure that each layer has a style/type associated with this.  Go to Tools--> Control Center (alt-c) select a layer, and then click Options.  Choose a type from the Classify Feature As drop down.  This is important, because you will get errors later saying that you have an unknown type = 0, and in the end breaking your installation of MapSource.

Step 3 Alternate:  The alternate method is to click the Edit tool.  Select all the features for one layer (you can use the Control Center to turn them off) and then right-click Edit Selected Features.  In the dialog, under feature attributes, choose New.  In this dialog use MP_TYPE for the attribute name, and then under value you need to enter the HEX code for that type, e.g. 0x01.  You can find different HEX codes in the cGPSmapper manual (C:\Program Files\cGPSmapper\cGPSmapper-UsrMan-v02.4.4.pdf    if installed in default location, page 113, 111).  It's much easier to use the previous other Step 3.

Step 4:  In Global Mapper, File --> Export Vector Data -- Export Polish Map.  Choose a name, and region for export.  Click ok and you will be prompted for a file location.  I've been saving mine under C:\Garmin\prjfolder.

Step 5:  Go to Start --> Run and enter cmd.  This should bring up a DOS prompt navigate to the cGPSmapper install folder.  cd c:\program files\cgpsmapper

Step 6:  Type cgpsmapper.exe c:\Garmin\prjfolder\filename.mp -o c:\garmin\prjfolder\20090618.img

Change the paths to reflect your file system.  The *.img filenmae needs 8 numeric characters.  I find using YEARMMDD format works best.

Press return and wait till it is done processing.  If you see the unknown type error, follow step 3 - 6 till it produces no errors.

Step 7:  Open the MapSetToolKit.exe from whereever you unzipped it.   Click Select IMG and navigate to c:\garmin\prjfolder\ and it should automatically find 20090618.img for you.  Click Add --> and it will add it to the right pane.  Click the elipse ... next to Mapsetdirectory and select c:\garmin\prjfolder\.  Give the mapset a name.  Under family ID enter 3 unique numerical digits.  Under cgpsmapper/cpreview folder, click the elipse ... and navigate to where the cgpsmapper.exe folder is (c:\program files\cgpsmapper\).  Check Install in Mapsource, and Blank Overview Maps.  Click Start.  It should say something like operation termined [sic], and ad to the list of Mapset installed.

Step 8:  Plug in and turn on your GPS unit if you have not done so.  Start Mapsource.

Step 9:  Go to View --> Switch To Product and pick your named mapset.  It should switch to this in the view.  Click the Map Tool button (looks like a polygon) and select the mapset.  Click the Send to Device button and follow the instructions.

Step 10:  Assuming you are in the area, go outside and test it.  If you only see outline boxes but not the map, keep zooming in until you see the presumably more detailed map that you downloaded.

It might in the end be cheaper and easier to just by something that does all this for you, if you don't have a license of Global Mapper.  However, you will get much more use out of Global Mapper as a tool for loading different datasets and reprojecting or converting than you would with something specifically taylored for Garmin GPS mapsets.  That is if you are a GISer or cartographer or use spatial datasets a lot.

 Hope that helps in some way or another.  I partially put that here so that I can remember how I did it :).

I've noticed more and more people talking about SQLite and in particular Spatialite.  This is typically surrounded by talk about the hopeful end of shapefiles as a defacto exchange format.  I suppose the success of shapefiles has mostly been because of the openess, so every proprietary GIS software was able to support it.  Now pretty much everything spatial is available in shapefiles.  In typical government fashion the U.S. Census Bureau is just now releasing their data as shapefiles instead of Tigerline files...that's why it seems appropriate to move to another format now.  Is there any greater sign of the death of technology when a government takes it on board?  When I worked on the FEMA map modernization one of my greatest frustrations was that they required ESRI coverages to be submitted with the final product.  Not only is ESRI stopping support for coverages, it requires an ArcInfo license...and even then you have manually add the coverages toolbox.  Of course, if you want to try and edit a coverage you have to go back into ArcInfo workstation.  Anyhoo, that's not relevant to this entry.  I decided to take a look at Spatialite and see what it was about.  First lesson learned - it's spelled with one L. 

So what is the beauty of this format?  It doesn't install ANYTHING. Even the gui interface can just be run from the exe file.  This makes it really really easy to get started.  So I created a simple spatialite db and imported some Open Street Map shapefiles which contained thousands of polygons and polylines to check out the stability.  I thought I'd start with some simple SQL commands and started with AREA for the polygon geometry.  Well this crashed the gui without any explanation.  So then I moved to IsValid (this uses the GEOS API) to see what was wrong with the polygon geometry, and this crashed the gui again.  I switched at this point to the commandline thinking the gui just couldn't handle the amount of data I was using.  At least when the commandline came up with an error it told me what the error was and where it was stopping.  The first thing I discovered was I had a dozen or so polygons that were not closed (start and end point did not match).  I extracted these and moved them to another table using this select statement:

CREATE TABLE buildings_err
AS SELECT * FROM buildings WHERE IsClosed(ExteriorRing(Geometry)) = 0
DELETE FROM buildings WHERE IsClosed(ExteriorRing(Geometry)) = 0

After this it still would crash if I ran IsValid.  So I tried this Select statment to extract all the polygons that had less than or equal to three points creating it:

INSERT INTO buildings_err SELECT * from buildings where NumPoints(ExteriorRing(Geometry)) <= 3;
DELETE from buildings where NumPoints(ExteriorRing(Geometry)) <= 3;

Finally I then removed all the invalid polygons:

INSERT INTO buildings_err SELECT * from buildings where IsValid(Geometry) <= 0;
DELETE from buildings where IsValid(Geometry) <= 0;

 

And then ran an itnersect with another polygon table (cleaned that one up as well):

 SELECT *, intersection(buildings.Geometry, natural.Geometry) FROM [buildings], [natural]

 This hung up on me too.  I'm not sure where all the polygon errors were coming from, either the shapefile, or the import has issues.  The GEOS api seemed pretty sensitive to these errors as well.  So I gave up on trying after it crashed and moved on.  One thing I discovered is the geometry column, and thus the table, can only store one type of Geometry.  This extends to MULTI as well.  So you have to separate your POLYGONs from your MULTIPOLYGONS or I guess explode the multipolygons to polygons.  I think that might get confusing...Note the David in this google group is not me.

Finally, after all this I thought to myself - "Hey wait a second, AIR uses a SQLite database!"  But the SQLite functionality is all built-in so you can't swap dlls, and actionscript doesn't reference external libraries unfortunately.  So I tried the handy built-in SQLite function load_extension, but this too is not apart of the AIR SQLite functionality!!  KAAAAAAAHHHHHNNNNNNN!!!!!   Sorry for the Star Trek reference.  So no go to use a Spatialite database with FLEX/AIR.  And it is sort of moot anyway because I don't know any commercial GIS that fully supports Spatialite.  I think FME does...ESRI certainly doesn't.  So thus was my foray into Spatialite.  Still think it is pretty cool, but I'm a visual person and need to see the data not just type in a few sql statements.

I would have probably given up this topic by now, but I was listening to the excellent Digital Planet the other day when it came up again.  It was an interesting take on it.  They are using volunteers ("non-experts") to classify and organize the hundreds of thousands of images of galaxies produced by the Hubble telescope (or others).  The organizer was originally worried that the experts would challenge the results, but he found that the volunteers actually worked better than an expert.  This was because the volunteers did not approach the problem with the same pre-conceived notions that an expert would.  This happened to coincide with something I was reading, where the author explained why philosophy has a fixation with jargon and creating new words.  Essentially it is the same, by using new words it allows the reader to step back and not approach the problem with all their pre-conceived notions.  I can appreciate this.  Just thought I would share.

Back in Colorado for the week.  Then headed to the AAG Conference in Las Vegas next week.

 Before I left, I needed to create some Service Areas using Network Analyst.  It didn't need to be exact but more of a visual example.  I started by calculating the length of each road segment using the geometry calculator.  Then I assigned average speeds to each type of road (e.g. 30 kmh for a 4wd road).  Then for travel time I divided the length (in km) by the speed to get the time it would take to travel that segment.  I also calculated this in minutes by multiplying the travel time in hours by 60.  Finally I created the network dataset using ArcCatalog, created a new service area layer then processed it calculating a 2 hour service area.  It kept stopping at 45 minutes and I thought I was doing something wrong, given that I had never done this before.  Turned out I had an overshoot at one the point where it would stop calculating.  The lesson - never trust someone elses data.  I should have run a topology before doing any of this, and saved myself a headache.

 I was jetlagged last night and woke up at midnight.  So I spent my time learning about Augmented reality.  I still dislike the term, because I feel that the data that supposedly augments reality is really part of reality anyway.  The wikipedia page as a good introduction.  It also had a link to a video about Wikitude Travel.  Now that is pretty cool.

 

Was thinking about the Seattle Examiner switching to online only.  We hear a lot about big brother references, but this one made me think of it again.  In 1984 Winston Smith's job is to alter newstories to fit whatever the government's current thinking is, new allies were always allies and never enemies, etc...At least with a hardcopy paper newspaper, it is harder to change the story.  But if everything is online, it is easy to change history because the newspaper stores the data.  Not really a conspiracy theorist, just thought it was interesting given the Seattle Examiner's shift and the NYTimes and Guardians APIs realeasing historical newstories.  What does this have to do with GIS - absolutely nothing...