SAU data to OSM

About

The urbanism and planning service of French Polynesia (or SAU, similar to National Geographic Institutes) asked A Vos Cartes to include their data into OSM, in order to facilitate collaboration with OSM community and improve data quality on OSM but also on institutional databases. This project is made of several parts : one for import, one other for integration and one for advising. This wiki page, only relates data import.

A Vos Cartes is run by Violaine Doutreleau (Violaine_Do (on osm, edits, contrib, heatmap, chngset com.)). She developed the Missing Maps project at CartONG (french mapping NGO) while she was in France. Now after 5 years of experience creating OSM data and using it, plus supporting OSM communities development (especially the one in Madagascar), she offers her GIS services to institutions, associations, in French Polynesia, through A Vos Cartes.

Goals

SAU hopes that, improving quality of OSM through this effort, will foster institutions into sharing their data, ease dynamic collaboration and support development of social and economic activity based on geographical data (such as delivery service, or accessibility projects).

Schedule

1. Data quality :
   1. checking data quality = done
   2. data quality improvement : SAU improves its data quality based on our recommendation and their knowledge (mainly tagging incoherence or incompleteness, or data updates)
2. Communication of import to be done by end of august (announcement and discussions)
3. Data preparation to be done by end of September
4. Import pilot (trial) to be done on a commune, between mid September and end of September 2019, if import is supported by the OSM community
5. Extend import to other communes (if process is accepted and validated) : to be done by end of December 2019

Import Data

Data description

SAU data gathers wide-ranging data (education, roads, land use, rivers etc) and specific data such as geodesic reference point.

Background

Data source : One must go at SAU office and ask for their data. French Polynesia is currently working on a data sharing platform. It is also possible to see maps services that render SAU data through the official map portal of French Polynesia TeFenua

Permission

1. Ministerial decree saying that SAU digital data are available under CC BY 4.0
2. Agreement of compatibility between CC BY and ODbL : this allows the integration of SAU CC BY 4.0 data into OSM, and rise the incompatibility parts

OpenStreetMap License compatibility yes see above.

Import type

One time import. In the future their might be updates but this is not yet planned, we consider it is a one time import.

Data preparation

Data reduction and simplification

Process and data selection are detailed here : SAU data to OSM/Data transformation and follow up

Tagging Plans

The tagging plans details are available here : SAU data to OSM/Data transformation and follow up

For information, we used :

• Map feature and status key/values to assume validity of a tag (approved or defacto).
• Taginfo, when no status or mapfeature pages where found, to assume the use of a tag,

As working at a local scale and no plan to export this to other places or countries we specify in the data mapping table the tags we plan to experiment when not approved by the global OSM community. We assume there is no risk of degrading the global OSM tagging scheme as working locally only. When in this case of experimenting we spot it in the data mapping.

Changeset tags

Here are the changeset tag we plan to use :

• comment=SAU data import to OSM, 2019
• import=yes
• url=https://wiki.openstreetmap.org/wiki/SAU_data_to_OSM
• source=SAU
• hashtag=#A_Vos_Cartes #SAU #name_of_imported_layer (ie worship) #2019
• accuracy=0.5 or description=avg accuracy is 0.5m

With the account avoscartes_import.

Data transformation

Transformation : We will code a SQL script to allow redo the transformation afterwards (as their might be potential updates).

Data transformation results

Data will be exported into geojson.

Results are here: https://ncloud.zaclys.com/index.php/s/T7bxeMcwmntoyx2

Data merge workflow

Team Approach

This import is made solo but supported by local communities (french polynesia), and with potential help from local OSM community.

Reference

This import is referenced in the Import Catalogue. Local OSM community (french + french polynesia) and import mailing list will be contacted and notified of import plans detailed here.

French and french polynesian communities have already been informed about the project.

Workflow

1. Data is splitted into 2 layers : for import vs for integration.
   1. Open validated and transformed SAU data in JOSM
   2. Download osm data in a new layer,
   3. Use conflation plugin and conflate OSM to SAU in order to split data into 2 separate layers : one for direct import (no need for merging as no OSM data in the area), one for integration (need for conflation / merging)
2. Data ready for import is
   1. added to OSM layer through copy / paste at source position (needs allow expert mode in JOSM preferences)
   2. validated with JOSM validation tool
   3. sent to OSM with changeset information detailed in the dedicated section, and avoscartes_import account.
3. Rest of data is validated and conflated manually

Data for French Polynesia is equivalent of data for a department in France (mean size of french departments), which lead us to not develop algorythm that handle conflation. We chose to do that part manually.

We will prioritize import per layer, with roads and buildings first as other layers depend on those.

Dividing up the work

Work to be done on a layer basis (buildings, road etc.) and :

1. One first trial on Arue, commune of Tahiti, French Polynesia.
2. Then depending on the size of the layer : commune by commune for Tahiti, island by island for the main islands, and archipelagos.

Changeset size policy

Max 10 000 element per changeset. JOSM can handle upload size so we will use this possibility if necessary.

Revert plan

We will document the changeset we did on the data mapping page, in case of a need of a revert.

Conflation

Conflation is done manually. Data that need conflation will be separated and integrated manually.

Quality control

Before importing we check data through JOSM validation tool (errors and warning).

After importing we check our import with OSMOSE QA, on edits made by our OSM account used for import (high and medium level).

On top of those generic approach, here are the points we added to our checklist :

• check retaining wall topological errors such as : do not cross buildings, and check if right directions
• check buildings shapes when buildings was removed (following SAU layer of deleted buildings)

Follow up

Follow up of import will available on wiki page data mapping, which details layer by layer the changeset and data that was imported.

See also

Mail to the mailing list :

1. Announcement of the project 2019-07-26 (in french) : https://listes.openstreetmap.fr/wws/arc/local-polynesie/2019-07/msg00004.html
2. Announcement of the plan 2019-08-31 : https://lists.openstreetmap.org/pipermail/imports/2019-August/006064.html
3. Discussion on ref system choice (in french), conclusion is to use local system as approximately similar to WGS84 (30 to 50cm with standard transfo in QGis RGPF>WGS84, 1.5 m when taking into account tectonic plate movement), and high precision of source data : https://lists.openstreetmap.org/pipermail/talk-fr/2019-September/094202.html Issue is : we would need to transform with plate movement but OSM is not going to update its data depending on the date so it will be punctual and if doing so, people are likely to not do the proper inverse transformation to get it into local system and so lose accuracy.