LINZ/Howto

From OpenStreetMap Wiki
Jump to navigation Jump to search
Logo.png
This page describes a historic artifact in the history of OpenStreetMap. It does not reflect the current situation, but instead documents the historical concepts, issues, or ideas.
About
Information
 This page was last updated 12 years ago and contains out-of-date information. See LINZ for the latest info.
Captured time
2012


A short summary of how to help with the OpenStreetMap LINZ data import

General

Getting started with JOSM and Merkaartor

There are many tutorials for the JOSM and Merkaartor editors elsewhere in this wiki, I won't try to replicate one poorly here.

The basic point is that you'll have to download one of the programs and install it locally first, and you will also need to create yourself a OSM login account to be allowed to upload data to the main OSM servers. If you don't have one of these yet, do it now:

Go to http://www.osm.org and in the top right of the screen click on "sign up"

So far we have been using the LINZ Data Upload login account to do the bulk uploads. Ask about this on the NZOpenGIS mailing list.

If you are just starting out and a bit hesitant to mess something up, you can try uploading to the "test" server instead of the "real" one. You won't be able to view it online, but it is good for practice. In JOSM go to Edit→Preferences Connections tab set untick the default OSM server and for the OSM Server URL type: 

http://api06.dev.openstreetmap.org/api

(?correct?)

Once you are confident that you are doing the right thing switch it back to use the "real" server & live database.

Getting started with the LINZ data

Using "Rob's method", described here:

The general idea is that you need to login to Rob's LINZ-2-OSM web app (ask Rob to set you up a login first)

  • Aug 2012: our new web app is now online, the following info needs updating! (although hopefully the new app is easy enough to figure out without too much in the way of detailed instructions)

once logged in you'll see a button on the front page: 

Convert:  [Run OSM Conversion]

then 

Bounds: [    ]   (minx,miny,maxx,maxy)
Dataset: NZ Mainland V16
Layer: Coastline

The bounding box bounds are given in x for degrees longitude and y for degrees latitude. Southern and Western hemispheres get a negative sign. Thus bbox=min_x,min_y,max_x,max_y is west,south,east,north

Some example bounds for various places around NZ are given in this wiki at LINZ#Areas.

  • todo: how to use OSM website/tools to create this bounding box for you. Displayed in JOSM/Merkaartor windows prior to download?

Uploading

Uploading high-resolution coastline

The idea is to replace the current PGS coastline with the much higher LINZ coastline data.

  • Log in and download a chunk of LINZ coastline as an .osc (.osm changeset) file from the LINZ-2-OSM site as above. Remember uploads must be less than 50,000 nodes at a time, so don't try to upload half the South Island in one go. 20-30,000 vertices at a time is about the most that is manageable. If you upload too many you'll get a ton of broken nodes which you will have to clean up manually before doing anything else.
  • In JOSM or Merkaartor download the area of interest.
  • Keep in mind that OSM line features have a direction to them, and that coastlines must be continuous, and are defined with land on the left side of the direction of travel. You might have to zoom right in to be able to see the arrow ticks in JOSM or Merk. (or switch them on manually in the preferences window).
  • Do File → Open to load the new coastline layer.
  • Review and explore what you see before you. Make sure your new coastline is better than existing coastline. Make sure the new data fits to what is there. Link up river mouths, beaches, rocks, etc. Compare with aerial imagery. Mix and match the data you have before you to get the best from old and new data.
    Note: Bing satellite maps are often very poorly aligned in New Zealand. Don't trust their imagery. LINZ on the other hand generally does quite a good job and in general you can trust them. Specifically, don't "fix" the LINZ data to match Bing imagery placement!
  • Select the old PGS coastline for removal. In JOSM do Edit → Search for the search string type :
    natural=coastline source=PGS
    This will select coastline features that are tagged as having come from PGS.
    Once the search is complete the found features will be highlighted in red. Type "3" in JOSM or F3 in Merk to zoom to them.
  • Edit → Delete
  • File → Upload data.
    • Specify a detailed commit comment like:
      Replace old PGS coastline with LINZ version: Southern Pitt Island
    • It's a bit more robust to upload in smaller sections, especially if you have a flaky internet connection. At the bottom of the JOSM upload page select the "Advanced" tab, select
      [*] Upload data in chunks of objects. Chunk size: 750
      n.b.: this has nothing to do with the 50,000 feature changes per upload limit and will not help get around that.
  • Finally when you are ready and have double-checked everything, click the [Upload Changes] button.
  • Coastline changes are handled differently than all other layers and can take up to 3 weeks to appear. (a world-coastline shapefile is produced from the data about once a month, and the rendering is based on that). So don't worry if you don't immediately see the change.

Uploading large polygon layers

  • Keep ways smaller than 2,000 nodes and greater changesets smaller than 50,000 nodes or the 0.6 API will get upset and you'll have a big mess of orphaned nodes to clean up!
  • Always run the JOSM Validator tool. The LINZ data is pretty good, but there are often some duplicate nodes and ambiguities to clean up. Aim for zero warnings and errors.
  • If you want to upload just a selection, if you simply draw a box around a bunch of features it only selects the nodes and ways. You can use the Search tool to also select all "type: relation" but that also ignores any sort of spatial selection and selects them wherever they are in the entire layer. Solution: draw a box around everything you want to select, then go into the JOSM search tool and do a search for "parent selected", with the "[*] add to selection" radio button pressed. This allows a selected node to also pull in its way, and all ways to also pull in any relation they belong to.
  • Relation cleanup: If you combine two outer-role ways with no inner-role present you'll get a left over empty way. When you Combine Ways you get asked to keep one and drop one outer role. Keep one, then in the multipolygon editing window click on the garbage can icon in the top of the window to drop the now unneeded relation.
  • Finally, in JOSM do File → Upload selection, and confirm all selected features. See the notes above about breaking the upload into a number of smaller chunks to minimize the effect of transmission/server problems (which are common).

Cleaning

Removing duplicate nodes

Background reading:

  • See LINZ#Fixing_duplicate_nodes_between_two_layers
  • Go to LINZ#QA_Maps in this wiki
  • "Duplicate nodes map" will show you where the dupes are.
  • Zoom in on some, check to see if they are candidates for merging.
    (Things like bridges crossing streams or roads parallel to forests should not be merged (as the forest does not actually make it into the centre line GPS coordinate of the road, and (with the exception of fords) the road and the stream are not actually connected), but things like sand-ground cover transitioning into mud ground-cover or rivers connecting to lakes should be merged.)
  • Open JOSM
  • File → Download from OSM
  • In the slippy map tab select a small area to work on and click the [Download] button at the bottom.
    After it downloads your map data should appear in the JOSM editor window.
  • Edit → Search
  • To search for e.g. lakes touching wetlands, search for "natural=water|wetland".
    (the '|' means "or", so add to the selection if the tag has natural=water OR natural=wetland)
  • With these features now selected, click the [Validate] button in the bottom right of the JOSM window. (if it's not there try enabling the Validator plugin in the Edit → Preferences menu's Plugin controls)
  • A list of errors and warnings appear in the bottom-right of the JOSM window. It is best to deal with duplicate nodes first, as that may make other errors (like overlapping areas) go away. (re-run the Validator after)
  • You can open the "[+] Duplicate nodes" list and right click on one of the pairs to zoom to the problem for further inspection.
  • If you are confident everything on the list should be merged, you can click the [Fix] button in the bottom right of the Validator results box to bulk fix them. Use with caution.
  • Alternatively you can left-click drag a box around a node, if there are two at that point you should see two or more nodes listed in the selection box on the right side of the JOSM window. Then Tools → Merge nodes. You should see the node box change appearance very slightly.
    (use with care as it is easy to merge two nodes which are not actually duplicates, just rightly close together)
  • Upload changes as above.

Mark II: Rob Coup's LINZ-2-OSM web app

April 2010

  • LINZ raw IFF format is loaded into PostGIS database with the linz_topo tool hosted in google_code.
  • Visit the web app site http://linz2osm.coup.net.nz/
    (helpers can ask on the nzogps mailing list or contact User:Rcoup at robert@coup.net.nz for a login)
    The base web app code can be found at linz2osm in github.
  • You then match the LINZ attribute with OSM wiki approved tags in the web app, details in the comments section, and then request the web app to create a .osm export file for you. If there is no matching OSM tag already use LINZ:attribute_name as the tag.
    • Be logical about adding things if needed (eg natural=tundra doesn't officially exist but it is pretty obvious that is probably just because of a lack of OSM effort in those parts). Do lodge new keys as a Proposal in the OSM wiki.
    • If it is as good as it's going to get for now, but there are lingering doubts, add the issue to the LINZ revisit list.
  • Import this .osm file into JOSM/Merkaartor/whatever and check for defects, correct tags, etc.
  • Use the JOSM Validator plugin to check for errors. Please email feedback to the nzopengis mailing list or file a bug at the github site so we can fix the problem at its root in the export scripts.
  • If you have doubts, trial upload it to api06.dev.openstreetmap.org, not the main live database.
  • Let the dust settle, see if it still looks normal in the morning.
  • After review & consensus that it is ok and finalized, upload to the main database.
  • Once uploaded, make note of this in the LINZ-2-OSM web app and the List of LINZ layers and notes wiki page(s).

Tips

  • Upload changeset sizes must be smaller than 50,000! I'm pretty sure that uploading in chunks of 750 or 1000 with JOSM's Advanced upload options (while helpful) will not help with that restriction.

Scripts

Bridges

or

  • Bridges almost align with road nodes, but not exactly. Roads are downloaded by JOSM with a precision of 6 or 7 numbers after the decimal point (~11cm; "%.9g"?), while bridges are delivered from LINZ-2-OSM with 9 or 10 (~.1mm; "%.12g"?). To merge we will have to round the bridge nodes to '%.9g' precision; then snap duplicate nodes in the JOSM Validator; then somehow spit the road at the end nodes of the bridge; then combine duplicate ways, merging tags using the JOSM Validator overlapping ways tool (which lets you know about them but doesn't fix them). ^%*!$#@!
  • Script for reducing the precision of nodes
    • todo: Wish for the Validator plugin to get a threshold parameter
    • todo: Wish for the Validator plugin to fix overlapping highways by safe merge

Troubleshooting

Stale tiles

  • Step 1: clear your web browser's cache
  • Step 2: make sure you are not behind a proxy server

If you only edit a tag and not the base spatial data the slippy tile may not get marked as dirty and so Mapnik won't know to rebuild it. If that happens you can right-click on the osm.org OpenLayers canvas and 'view image' to view just the one tile. then in the address bar add /status to the end of the png's URL to see the last time it was rebuilt. add /dirty to the end of the URL to get it rebuilt the next time someone views it. (There will be a slight delay lag before this happens, so be patient and you also probably have to bypass your web browser's cache (shift-reload) to see the new version). /dirty is only meant for limited development debugging, use sparingly.

"Leaky" Osamarender tiles

  • Note: As of mid-2012 Osamarender is offline

While trying to clean up some Coastline I notice that in osm.org's Osmarender that some tiles are "flooded". In a manual audit + JOSM's Validation tool I don't find any obvious errors, but perhaps I missed something. The osm.nl coastline checker seems to be offline since last April and the osm.org Mapnik coastline is too long out of date to be useful feedback. I read somewhere that for osm.org's Osmarender there is a file in svn somewhere which states which tiles are wet, which are dry, and which are mixed. Is there a way of seeing (maybe an OpenLayers overlay) what a particular tile is classified as? so then a request could be filed somewhere to get its wet/dry/mixed status updated? Or maybe it just takes a while for the renderers to sort themselves out?

examples of the breakage:

Fixing duplicate nodes between two layers

Some layers, like roads+tracks or fences+hedges will naturally connect be be provided in two separate source layers. Where they meet their nodes should be merged. The trick here is to only merge duplicate nodes between layers which really need it- for example connecting a road network to a fence line might lead to some unfortunate turn-by-turn direction routing one day!

Single key, multiple values

Using JOSM to download

(clustered features)

  • Download an area to work on in JOSM
  • Use the search tool to select e.g. "natural=wood|water|mud|sand|wetland|scrub|heath" by replacement
Using Xapi to download

(sparse features)

The Xapi download interface can be used with the wget program to download just the two layers we are interested in. Those can then be uploaded into JOSM and duplicates found using the JOSM Validator plugin, merge them, then upload the changes. The data in this example will be fences and hedges in the Chatham Islands.

  • Make sure both way layers are already uploaded.
  • Download with wget + Xapi:
wget -O chatham_hedges_and_fences.osm \
 "http://osmxapi.hypercube.telascience.org/api/0.6/way[barrier=hedge|fence][LINZ:source_version=V16][bbox=-177,-44.5,-175.5,-43.5]"
If way layers are already uploaded you must re-download them from the server.
  • Import this .osm file into a new JOSM session. (don't download anything else)
  • Zoom out a little, then drag a box around everything to select all
Fixing with the Validator
  • Make sure the Validator controls are running (tick-mark button on left, text window in bottom right)
  • With area of interest selected press the [Validate] button (bottom right)
  • Expand the error tree in the validate controls by clicking on the little sideways lollipop
  • Expand and click on "Duplicate nodes" to select+highlight them
  • Investigate, confirm (click on map canvas to unselect all, right click on error list item & 'zoom to error')
TODO
Verify tags are preserved (eg gate nodes into roadways)
  • Press the [Fix] button (bottom right). This only acts on items selected+highlighted in the error list.
This may take care of a bunch of other Validation warnings. So start on the duplicate nodes first, then re-validate!
  • Check the command list changes
  • Upload to the main server
  • Done!

Xapi takes a while to get updates from the main data server, so when using it you need to wait a little while before re-downloading something you've just uploaded.

Multiple keys

This is the same as above but you have to run Xapi twice and import each .osm file individually. If merging nodes to ways (for example gates to roads) it might be possible to proceed with only the roadways already uploaded (untested). If that works it will be the preferred method.

If you try and merge two layers not yet uploaded to the main server it is likely that the upload will get confused if it tries to modify nodes on the server which do not exist yet. In these cases you might export the locally modified .osm files to a new merged one, then close JOSM and then reopen with the new single merged .osm file. Then upload that. (untested, and more dangerous). Generally to aid with fixing mistakes you will want to upload the raw data in one step, then fix it in another. That way if the fix has to be reverted it is possible to do so. (aka "atomic commits").

Undoing uploads

  • You can use Xapi to download a .osm file with just the selected changeset ID. Download all the features from that changeset (and only that changeset!) using Xapi, open it into JOSM/Merkaartor, delete everything, then upload changes. untested - use care
 wget -O changeset_4593253.osm \
    "http://osmxapi.hypercube.telascience.org/api/0.6/*[@changeset=4593253]"

Be quick! If anything has changed since the upload this is likely to break.

  • There are several OSM Revert scripts around the wiki to have a look at.

Fixing bulk tagging mistakes after upload

(try not to get here)

  • You can use Xapi to download a .osm file with just the selected query attributes. Set the bounding box (bbox=w,s,e,n) term to constrain it to a certain geographic area. For example if you wanted to change all features with access=unknown to access=private in the Chatham Islands you could run:
wget -O chatham_access_unknown.osm \
 "http://osmxapi.hypercube.telascience.org/api/0.6/*[access=unknown][LINZ:source_version=V16][bbox=-177,-44.5,-175.5,-43.5]"
  • Next open this .osm file in your OSM data editor but do not download more.
  • In Merkaartor do:
    • Right click on the download.osm layer and Zoom to it
    • Edit → Find
    • Set Key to "access" and Value to "unknown" and hit [Ok]
    • A list of matching features should be selected in the bottom-left of the program window. In the middle of the left side is a list of shared tags.
    • Change the tags as needed and then review
    • When you are ready Upload the changes.
  • In JOSM... ??

See the Xapi help page for information on how to make more complex or specific queries.

Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=LINZ/Howto&oldid=2581805"