Ramani Huria OMK Field Data Cleaning Workflow

Preparing for OMK field data collection

General documentation from the OpenMapKit website.

1. Develop data model.
   • Examples for Tanzania (see Dar es Salaam/Ramani Huria), Uganda, Liberia.
2. Create ODK (OpenDataKit) and OMK (OpenMapKit) forms based on step 1 - the Data Model.
   • More documentation on xlsform.
   • Conversion from .xls to xml: form format conversion
   • Example forms are available from Uganda and Liberia.
   • Constraints on (specific) questions can be applied by creating/adapting the constraints/default.json file (found in the OpenMapKit installation directory on a phone), according to the specifications about Constraints in OpenMapKit documentation.
3. Get a base map (through humanitarian map, or any other way to create an mbtiles map. Such as tl, or Tilemill). An online map can also be used, if connectivity is good.
4. Get base layer OSM data (either from the export tool, Overpass turbo or through JOSM) in .osm format
   • If through JOSM:
     • Make sure it includes feature types you’d like to be able to edit through the forms.
     • ‘File’ > ’Download from Overpass’.
     • Select proper area, use query: building=* OR amenity=* OR office=* OR shop=* OR man_made=* OR highway=* OR waterway=* (or applicable parts of that query).
5. Install ODK Collect and OpenMapKit from the Google Play store for an Android mobile device.
6. On the mobile device:
   • Copy forms to odk/forms, the base map to openmapkit/mbtiles, and the OSM base layer to openmapkit/osm.
   • If custom constraints are used, copy to openmapkit/constraints.
7. Off you go... fill in some forms.
8. Once data collected, grab data from the phone; copy odk/instances.
9. Grab all the .osm files, pull into JOSM, merge layers, check data, upload.

Field data cleaning procedure

After field data collection with OMK, clean the data before .

Step 1. Create folders

Create the folder structure for the data cleaning process:

1. Create a primary working folder: omk_field_data_cleaning.
2. Inside it, create a container subfolder: ward_name_current-date.
   For example: sinza_2017-09-22.
3. Inside it, create container subfolders:
   • Add a folder for the raw data: /01_raw/yourOSMusername.
   • Add a folder for the working data: /02_working/yourOSMusername.
   • Add a folder for the final data: /03_final/yourOSMusername.

Step 2. Raw files

Your supervisor or team leader will download data from the server, and select sections of data for you to clean up.

Note to supervisor:

1. While dividing out this data to the different team members, purge everything else from except what you want to give to a specific team member, and then ‘Save As…’ that section. Do not copy a section of the data downloaded from the OMK server into another new file, as this will not sync while downloading data from OSM database to start cleaning field data. Read about the purge action.
2. Do this for every team member you are giving data to clean.
3. You will need the time-dependent filter to select data on the server over a particular period and give that to the team for cleaning. See some examples.
   • timestamp:2017-09-22/ - Will find any object that was modified after date.
   • timestamp:2017-09-22/201-09-25 - Will find any object that was modified after date but before other date.

For team members:

• The file your team leader gives you will be named as: ward_name_current-date_raw/yourOSMusername_raw.
  For example: sinza_2017-09-22_raw_lusdavo.osm.
• Copy the file your team leader gives you to the ward_name_current-date/01_raw folder. Format dates as YYYY-MM-DD.
  For example: sinza_2017-09-22/01_raw/sinza_2017-09-22_raw_lusdavo.osm.

Step 3. Add OSM files to JOSM

• Add all the .osm files to JOSM.
• Drag all the .osm file into JOSM.

Step 4. Sync field data with OSM data

Do a series of checks on the data to see if there is any issues with the data.

You will be using several filters during your cleaning process. Add the following filters:

1. type:node untagged
2. new OR modified

To prepare your data for cleaning:

• Select collected data using the JOSM filter: type:node untagged
• Select all results (Ctrl+A).
• Add the selection to the TodoList plugin: on the plugin, click ‘+ Add’.
• Deselect results by clicking in the Map View.
• Download OSM data in your area, to ensure we merge properly with existing data.
  • Press 2 on the keyboard to zoom to the extents of your collected data.
  • Select ‘File’ > ‘Download in current view’
• Now save this file to your working folder
  • ‘File’ > ‘Save as...’ > ward_name_current-date_raw/working_yourOSMusername.
    For example: sinza_2017-09-22_working_lusdavo.osm.

Step 5. Cleaning data

• You are now cleaning the ‘working’ file in JOSM.
• Add Imagery being used to clean data.
• From the ToDoList, inspect each feature (select the first one in the TodoList):
  • Are the locations correct?
  • Are there any spelling mistakes?
  • Are the tags capitalised correctly?
  • Are the tags fields filled correctly?
  • Are we not duplicating existing OSM data?
• If okay, click ‘Mark’ in TodoList to proceed to the next item.
• Add a filter new OR modified.
• Add imagery.
• Uncheck the type:node untagged filter.
• Select only new or modified data using the JOSM filter: new OR modified, and check if everything looks okay.
• Run ‘Validation’ in the ‘Validate Results’ window.
• Fix any ‘Errors’ and ‘Warnings’ that may be found in the data you have been cleaning.

Record errors encountered

Capture common problems that come up

• Team members should record any errors and problems encountered (both while mapping and validating).
• Please record this in a text file with screenshots if possible.
• Save to ward_name/current_date/02_working.
  For example: sinza/2017-09-22/02_working folder.

Final data check

Double check the validated data:

• Team leaders are to review their teams data validation.
• If satisfied, get the group supervisor to double check the data.

Upload data

If the sroup supervisor authorises it, upload the validated data to OSM:

• Add your OSMUsername and password to the JOSM preferences.
• Upload adding the #RamaniHuria #MissingMaps comment, and specify Survey as the datasource.

Step 6. Save final files

Save the final loaded layer file to the external hard drive

• Right-click the final layer that was uploaded and select ‘Save as...’
  • Save to ward_name/current_date/03_final.
    For example: sinza/2017-09-28/03_final folder on external hard drive.
  • File name: sinza_2017-09-28_final_lusdavo.osm