LV:LVM-import

From OpenStreetMap Wiki
Jump to navigation Jump to search

Step 1 of the import is FINISHED!

Goals

LVM (Latvian State Forestry) publishes a lot of useful geodata under CC0, on the Latvian open data portal.

Currently, me (osm user Vladekk ) made a plan and a script to import road data from LVM, from this dataset:

https://data.gov.lv/dati/eng/dataset/as-latvijas-valsts-mezi-mezsaimniecibas-infrastruktura

This can be done using Hootenanny conflation software and custom script to run it.

Import Data

Up-to-date data url can be found in the GitHub repo script.

Schedule

Import is planned to be executed during September 2021, finished till September 20th.

First step is doing proof of concept using one conflation grid cell. (Done) Then, after community approval, whole Latvian territory can be imported.

User for import will be

LVM_Import

buckler-bronx-xmas

(above is pa..wd)

Tags

Full translation scheme and tags can be found here: https://github.com/Vladekk/LVM-import/blob/main/lvm.js


Tag source will be set to source=LVM:XXX where XXX will be dataset name. In case of roads, LVM meža ceļi

name=ROADNAME from shp attribute.

name:forestry_district=LVM forestry district name (from LVM_DISTRI code and supporting docx file)

ref:forestry_district=LVM forestry district code (LVM_DISTRI)

operator=Latvijas valsts meži

operator:abbr=LVM

operator:website=https://www.lvm.lv

operator:wikipedia=https://lv.wikipedia.org/wiki/Latvijas_valsts_me%C5%BEi

surface: is set to unpaved for most ways, set to gravel to named ways. IMO, good compromise to have it usable. I expect most LVM ways to be gravel, but it is hard to tell.

Hootenanny

Hootenanny (Hoot from now on) is conflation software supporting wide range of conflation types and objects. For our import, experiment show best settings are ReferenceConflation with these options:


-D search.radius.highway=20 `

-D match.creators="hoot::HighwayMatchCreator;" `

-D snap.unconnected.ways.review.snapped.ways=true `

-D conflate.post.ops+=UnconnectedWaySnapper `

-D map.cleaner.transforms-=hoot::UnlikelyIntersectionRemover `


Hand-written translation script LVM->OSM


Experiments also show that changeset derivation in Hoot produces strange results in our case, possibly introducing errors in OSM. So, we won't use it.

Legacy workflow (DO NOT USE)

So, current plan is as follows

0. User chooses work grid cell they are going to work on (more below)

  1. Custom script downloads fresh data from LVM and converts to OSM format
  2. Script splits LVM data into 8x8 grid
  3. Script downloads fresh data from OSM for the current cell
  4. Script runs conflation on current cell using Hoot and options above
  5. Script marks modified ways and nodes as changed in the resulting OSM file, so JOSM can understand they need to be uploaded.
  6. User then can download and review OSM file in JOSM
  7. After review and manual fixes (mostly ways over ways and ways ending near other ways) user uploads imported cell to OSM

Cell can be chosen by the user who will validate and perform conflation. There can be possibly more than one user.

As this export is limited in scope, we can skip using Task Manager and just coordinate work using Telegram and email.


Current workflow for the coordinator

  1. Custom script downloads fresh data from LVM and converts to OSM format
  2. Script splits LVM data into 8x8 grid
  3. Data is converted to geojson
  4. Task is created in MapRoulette

Current workflow for the mapper

1. User chooses work grid cell they are going to work on from MapRoulette, downloads task data

2. Open LVM geojson as a layer in JOSM ("import layer")

3. Download existing OSM data as a new layer

4. Go road-by-road

A) In case highway does not exist, just copy it to OSM layer and join intersections to existing highways

B) In case highway exists and geometry differs, delete existing highway, copy from import layer, join intersections to existing highways

C) In case highway exists and geometry is very similar, copy all tags from highway on import layer, and paste to the highway on OSM layer

Notes:

  1. Make sure you only work on the same highways in both layers! If they end and start in different places, split highways and work on them part-by-part!
  2. Make sure you never delete highways with relations! In the case highway in OSM layer participates in relation, always just copy tags from highway in the import layer


Common issues and solutions

JOSM validator will list some issues and you should fix them before uploading changes.

1.Way over other way

A) Select way, press G to unglue, select "apply tags to both ways".

B) Press del to delete way a the top (selected). Make sure another way is left in place of it, that was under the first way.

2. Way ending near other highway.

A) Verify that nearby (Target) way has a node. If not, add it using key A.

B) Drag conlfated way (source) to this node and attach by pressing M

3. Duplicate node in a way or duplicate relation

A) JOSM fixes automatically. Press "Fix" in validations panel.


There can be others, like way connected to a building, more complex overlappings ways, etc. Use your best judgement.

Do not hurry. If in doubt, leave this specific cell to the more experienced mapper. Be careful to not harm relations. Do not leave cell hanging unfinished for long, euther finish and upload or leave it to others.

Example

Example files are here:

https://github.com/Vladekk/LVM-import/tree/main/examples

Files produced by script, in the order of generation

1.import dataset cell

import-gridded-00001.osm

2.openstreetmap source cell

osm-source-00001.osm

3.conflated data

conflated-00001-UnifyingAlgorithm.conf-ReferenceConflation.conf.osm

4.changeset of conflated data generated by Hoot, useful to understand, where changes happened. Dot not upload directly, use file below.

conflated-00001.osc

5.same file as with conflated data, but changes marked for upload and debug tags removed

changes-conflated-00001-UnifyingAlgorithm.conf-ReferenceConflation.conf.osm

6. This file is not generated, but created by user. You can use file above and just save your changes, but it is here in examples folder to make things more transparent. In this file, conflated data is validated and manually corrected.

validated-conflated-00001-UnifyingAlgorithm.conf-ReferenceConflation.conf.osm


QA

The email to the Imports mailing list can be found in the archives of the mailing list at https://lists.openstreetmap.org/pipermail/imports/2021-August/006704.html

Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=LV:LVM-import&oldid=2237225"