Kerala Road Import
1. Overview
In late July 2018, severe flooding affected Kerala state in India due to unusually high rainfall during the monsoon season, Kerala's worst floods in nearly a century over 373 people died within a fortnight, while at least 280,679 people were evacuated, rescue operations are going on. kerala floods wikipedia
Maps used in disaster response https://www.microid.in/keralaflood/ which is used to identify flooded roads and https://keralarescue.in/map/ , https://process.keralarescue.in/rescue used to visualise rescue requests are using OSM data.
In response to the disaster mapping efforts Facebook have shared machine learning based road data for Kerala, and offered help of their mapping team to add this data to OSM.
The plan is to use the help of FB's mapping team to import using their internal tools that takes care of conflation, connectivity fixes, etc.
will be following the workflow described on
https://wiki.openstreetmap.org/wiki/AI-Assisted_Road_Tracing
With Indian road tagging scheme: https://wiki.openstreetmap.org/wiki/India/Tags/Highway
The whole import be validated by community on tasking manager.
To avoid conflicts, will be archiving current active HOTSOM roads projects and setup a new project on tasking manager.
Discussions on various channels are collected on this github issue: https://github.com/osm-in/mapping/issues/1
OSM Imports email list discussion: https://lists.openstreetmap.org/pipermail/imports/2018-August/thread.html#5661
2. Schedule
With the help of facebook's mapping team, we plan to start the import by the last week of August.
3. Data Processing
The data has been made available as osm files for review.
https://fb-public.box.com/s/ggo37k7bvj92jj137wr4hpmd5ba5rjt1
Submission Process
The submission process consists of the following steps summarized in the image below.
(1) We use an instance of the OSM Tasking manager to divide up area and create a flow of editing and validation for the Facebook mapping team.
(2) The Mapper picks a task and loads an .osm/.xml file generated from the post processing step into an enhanced version of iD (this includes both the roads generated by Facebook and the current OSM data).
(3) The Mapper evaluates the roads generated (styled in a different color) and inspects them for issues like crossing highways, disconnected roads, incorrect intersections, short road stubs, road types, etc., and fixes these issues manually one by one, using DigitalGlobe satellite imagery.
(4) The modified iD tool is equipped with data validation functionalities similar to JOSM and osmlint. This allows the mapper to check for quality and conflicts with current OSM data prior to submission. Some examples of what we check for include:
- Isolated roads that are not connected to rest of the map.
- Roads crossing waterways.
- Roads with repeating node IDs.
- Roads with possible connections after a short extension.
- Self-intersecting ways.
(5) In case of conflicts between newly added roads and other OSM contributors' edits, our mappers will almost always choose “keep existing edits.” If we end up changing another mapper's edits, (for example, to align better with the latest version of DG satellite imagery) the mapper will leave detailed notes to explain why.
(6) Until all errors are fixed, Mappers will not be able to upload to OSM.
(7) After fixing all the issues detected in the iD tool and a visual review, the Mapper clicks the “Save local” button to save their edited roads locally for validation.
(8) A Validator then goes into the same task to verify that the data is correct by re-validating the data, makes necessary changes as needed, and clicks “Submit” to finally upload the new road data to OSM.
(9) The Validator then conducts a post-submission review of the submitted edits in JOSM. If any errors are detected, the validator addresses the errors and submits necessary changes. A custom JOSM paint style is used to help distinguish submissions from the Facebook mapping team from community submissions, so that all roads with the import=yes tag are highlighted in green. Some examples for what we check for include:
- Short Unclassified ways (between 10-60 meters)
- Highway crossing waterway without a bridge or ford tag
- Highway crossing buildings
- Overlapping ways
- Unspecific highway tags
(10) The Facebook mapping team will continue to monitor all comments made on their edits and reply to them in a timely fashion. The Facebook team is happy to incorporate feedback and local expertise into their validation process and future mapping projects.
4. Data Preparation
Highway Tagging values
During the submission process, mappers will assign and fix highway types according to the general highway tag guidelines of OSM http://wiki.openstreetmap.org/wiki/Key:highway and the India specific application of highway tagging values found in the wiki https://wiki.openstreetmap.org/wiki/India/Tags/Highway
Some other methods the mappers will use to classify roads include:
- Refering to local Wiki guide for India.
- Looking at existing used tags in the local area of mapping.
- When connecting to current OSM roads on the map mappers try to use the same tag where it makes sense to do so.
- Direct involvement of the local Kerala community for advice.
Changeset Tags The following tags will be used:
Changeset Tags:
Element Tags:
- import = yes
The 'source' tag indicates that road geometries are based on DigitalGlobe satellite imagery.
5. Data Merge Workflow
Team Approach
By request of the local OSM India community, this import (data integration) will be done through an instance of the OSM Tasking Manager, by Facebook engineers and a group of selected trained map editors. Among the skills required:
- Good experience with both iD and JOSM.
- Familiarity with OSM editing norms: how to merge nodes (M), join ways (J), combine ways (C) and un-glueing (G)
- Proper conflict resolution with both the custom iD tool and standard JOSM.
Our map editing team consists of Mappers and Validators, and their accounts are listed as below.
Local Team
Facebook Team
Mappers: Rena VLD017, Tina VLD020, Gerum VLD052, Linzi VLD053, Ryan VLD054, Jesspher VLD062, Louis VLD027, Savanna VLD033, Katherine VLD035, Olympia VLD036,
QA Team: Jeff RVR002
Facebook mappers are not bots :) The usernames (VLD001, RVR001, etc.) were made to help their internal process where they have paired Mappers and Validators to ensure quality. Please feel free to send them questions individually about edits or email all of them at osm@fb.com.
Partner Teams
Humanitarian OpenStreetMap Team (HOT) Indonesia
We have partnered with HOT's Jakarta, Indonesia office who are helping fulfill the import process: Adiatma IRM-RV, QA Hanif IRM-RV, QA Yeni IRM-RV, QA Akrimullah IRM-RV,QA Adam IRM-ED, Mapper Silvia IRM-ED, Mapper Dary IRM-ED, Mapper Irwan IRM-ED, Mapper Riyadi IRM-ED, Mapper Zainab IRM-ED, Mapper Deo IRM-ED, Mapper Anisa IRM-ED, Mapper atmahadi IRM-ED, Mapper Dennis IRM-ED, Mapper Fatri IRM-ED, Mapper Muty IRM-ED, Mapper Iko IRM-ED, Mapper Aglis IRM-ED, Mapper Diana IRM-ED, Mapper Fadlilaa IRM-ED, Mapper Arma IRM-ED, Mapper Elisabeth IRM-ED, Mapper Diah IRM-ED, Mapper
Workflow
In summary, our overall mapping workflow is:
(1) Facebook extracts grey-scale road mask images from high resolution DigitalGlobe satellite imagery. (completed)
(2) Facebook generates road vectors based on the road mask images and merge them with latest OSM data to produce .osm files. (completed, see link above for the data generated)
(3) The Facebook human mapping team does manual validation on the .osm files through their enhanced iD tool and in JOSM prior to submission (as described in the “Submission Process” above).
(4) After conflict resolution, submission, and post-submission review, we work with local Kerala OSM community to complete local validation as described in the “Validation Steps” below.
6. Validation Steps
There will be a 4 step validation process done by experienced OSM editors including local Kerala Mappers.
- We start with human validation of the generated roads by the Facebook Mappers as explained in the “Submission Process.”
- Before submitting the changes, the team will verify that their updates are based on and compliment the current OSM data.
- After Submission (as described in the “Submission Process” above) a team of selected Validators and QA's conduct a second post-submission review by passing over chunks of multiple submitted tasks, area by area, by repeating the JOSM validation checks in described in Step 9 under “Submission.”
- The validation process primarily only includes elements that the Facebook team has contributed, which can be identified by the “import=yes” tag. Elements that the Facebook team “inherits,” meaning elements that are authored by one of the Facebook usernames but lacking the “import=yes” tag, are not typically included in post-submission validation. For more about inheritance, see footnotes.
- We understand inherited versus contributed elements by exploring the ownership of nodes and their parent ways. We can identify inheritance based on authorship and tagging. Roads that are authored by one of the Facebook team, but do not include the import=yes tag, is likely an inherited way. Because we are a team of remote mappers, using one data source, we want to avoid overstepping our contributions.
- Community feedback on imported roads is evaluated and incorporated for final edits in this stage as well. All edits can be identified with the changeset tags above and the Facebook team will respond to any e-mails or concerns about edits within 48 hours.
Apart from validation by the facebook team, import will be validated by the Kerala community on the Tasking Manager.