Digitransit/Helsinki region Swedish road names import

From OpenStreetMap Wiki
Jump to navigation Jump to search

Helsinki Regional Transport Authority HSL has a need for the Swedish road names in OSM. There would be basically three possibilities to import the road names from: Digiroad Database (CC BY 4.0) from Finnish Transport authority, VRK Database (CC BY 4.0) by the Population Register Centre, or the Topographic Database (NLSFI License) from the National Land Survey of Finland. Since Finnish Transport Agency is partner in the Digitransit project and because Finnish Transport Agency has provided explicit written permission regrading the CC-BY 4.0 license that OpenStreetMap Foundation requires (guidelines) this plan considers the Digiroad option.

Goals

Aim of the import is to be able to show the Swedish road names in the various HSL maps. Therefore the practical goal is the following:

  • Import name:sv=* values for the roads in Helsinki, Espoo, Vantaa, Sipoo, Kauniainen, and Kirkkonummi municipalities that have both Swedish and Finnish as official languages (Finnish being majority).

Schedule

Plan is to do the import during 30th of March - 31st of March, 2017.

Following topics need to be considered:

  • Licensing rights (CC-BY 4.0), OpenStreetMap Foundation has now guidelines. Written permission by Finnish Transport Agency has also been received
  • Data preparation, i.e. how to prepare the Digiroad data for the import
  • Data merge, i.e. how to merge the prepared Digiroad data to the OpenStreetMap

The last two steps are explained in more details below. The actual import, after community buyout and license compliance is verified, can be done in about two weeks, one week for technical implementation, and second week for the actual import.

Import Data

Background

Provide links to your sources.

Data source site: http://www.liikennevirasto.fi/web/en/open-data/digiroad
Data license: http://creativecommons.org/licenses/by/4.0/ (see also http://www.liikennevirasto.fi/web/en/open-data/terms-of-use)
Type of license: CC-BY
Link to permission: http://bit.ly/2nvrxgk
OSM attribution: https://wiki.openstreetmap.org/wiki/Contributors#Finnish_Transport_Agency
ODbL Compliance verified: yes

OSM Data Files

Link to your source data files that you have prepared for the import - e.g. the .osm files you have derived from the data sources.

Import Type

Identify if this is a one-time or recurring import and whether you'll be doing it with automated scripts, etc.

New roads are constructed and their geometries added to the OSM also after the initial import, so import should be recurring but it needs to be find out how often this should be done.

Identify what method will be used for entering the imported data into the OSM database - e.g. API, JOSM, upload.py, etc.

JOSM will be used for entering the changed data to the OSM database.

Data Preparation

Data Reduction & Simplification

Describe your plans, if any, to reduce the amount of data you'll need to import. Examples of this include removing information that is already contained in OSM or simplifying shapefiles.

The name:sv=* tags will only be added to the existing ways that have the highway=* tag, no geometry additions or changes will be made. If a road already has the name:sv=* tag then it is not changed.

Tagging Plans

Describe your plan for mapping source attributes to OSM tags.

The roads with the highway=* that have name=* will be updated if a Finnish match for the name=* is found from the Digiroad database. The values of the field tienimi_su from the Digiroad DR_LINKKI_K.shp files are compared to the name=* in OSM. The values of tienimi_ru are exported from the Digiroad DR_LINKKI_K.shp and name:sv=* tags with the values are added in OSM.

In addition source:name:sv=Digiroad.fi is added for each modified road.

Changeset Tags

Describe how you'll use changeset tags in the import.

Data Transformation

Describe the transformations you'll need to conduct, the tools you're using, and any specific configurations or code that will be used in the transformation.

PostGIS database is used for storing the Digiroad data. OSM data is downloaded via the Overpass API. A Python script available in GitHub is used in the transformation.

Data Transformation Results

Post a link to your OSM XML files.

https://github.com/ernoma/Digiroad-Digitransit-OSM-import/tree/master/Digiroad_osm_files

Data Merge Workflow

Team Approach

Describe if you'll be doing this solo or as a team.

References

List all factors that will be evaluated in the import.

Workflow

Detail the steps you'll take during the actual import.

Information to include:

  • Step by step instructions
  • Changeset size policy
  • Revert plans

The Digiroad Esri shape files for Uusimaa county are downloaded via Digiroad download service and stored in a PostGIS database. The commands are as follows: 

sudo -u postgres -s
createdb -U postgres digiroad_import_db
psql -p 5432 -d digiroad_import_db
CREATE EXTENSION postgis;
CREATE USER osm_import WITH PASSWORD 'osm_import';
GRANT USAGE ON SCHEMA public TO osm_import;
GRANT SELECT ON kunnat_ita, kunnat_keski, kunnat_lansi TO osm_import;
\q

cd Digiroad/UUSIMAA/UUSIMAA_1
shp2pgsql DR_LINKKI_K.shp public.kunnat_lansi > digiroad_lansi.sql
psql -p 5432 -d digiroad_import_db -f digiroad_lansi.sql
cd Digiroad/UUSIMAA/ITA-UUSIMAA
shp2pgsql DR_LINKKI_K.shp public.kunnat_ita > digiroad_ita.sql
psql -p 5432 -d digiroad_import_db -f digiroad_ita.sql
cd Digiroad/UUSIMAA/UUSIMAA_2
shp2pgsql DR_LINKKI_K.shp public.kunnat_keski > digiroad_keski.sql
psql -p 5432 -d digiroad_import_db -f digiroad_keski.sql

One municipality at time, the municipality borders are retrieved via Overpass Turbo and exported as GeoJSON, e.g.: 

[bbox:{{bbox}}];
(
  relation[boundary=administrative][name=Espoo]
);
out meta;
>;
out meta;

Then following commands are run on command line: 

ogr2ogr -f GeoJSON -explodecollections espoo_rajat_union.geojson espoo_rajat.geojson -dialect sqlite -sql "SELECT ST_Union(geometry) as geometry FROM OGRGeoJSON"
python get_poly.py espoo_rajat_union.geojson espoo.poly

Then highways without Swedish names are get for the municipality via Overpass Turbo and exported as OSM files (the poly in the Overpass query is given contents of the get_poly.py script output): 

way[highway][name][!"name:sv"](poly:"");
out meta;
>;
out meta;

Then a Python script retrieves Digiroad road names from the PostGIS database, does the name comparison for OSM data and when appropriate adds the name:sv tags to the OSM data. The modified data is saved as an OSM file that is in JOSM format. The OSM file is opened in JOSM, validated and uploaded to the OSM database.

There are 10,000-15,000 ways tagged with the highway=* that have name=* but no name:sv=* in Helsinki region. The maximum edit number is 10,000 per changeset. Since the import is done municipality by municipality the changeset size is not an issue. If a municipality has too much data (maximum of few megabytes per municipality) for JOSM to handle then the import for the municipality is divided for a few smaller size sets.

If there is need for a revert then community is consulted, for example, via IRC: irc://irc.oftc.net #osm-fi.

Conflation

Identify your approach to Conflation here.

  • If a road already has the name:sv=* tag then it is not changed. However, a report of differences in Swedish names is created.
  • Roads at the borders of the municipalities are manually looked through before import. If a road in OpenStreetMap database continues over a municipality border and it is not yet split at the border then it is split. This is done because the name of the road might be different in different municipalities.

QA

Add your QA plan here.

Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=Digitransit/Helsinki_region_Swedish_road_names_import&oldid=2735091"