Bellingham Washington Import
Pending Acceptance
This is a proposal to import buildings and addresses for Bellingham, WA
About
This page is intended to document plans for an upcoming import of data from the City of Bellingham, WA
We plan to follow & update the plan found at import checklist, but our general high level plan is this:
- Identify data to import
- Translate, tag, and otherwise tenderize the data
- Assemble a team of locals
- Work together on a plan
- Make sure the community - OSM (imports@), OSM-US (talk-us@), and local mappers - are all on board with the plan.
- Train the team
- Divvy up the work using the HOT Tasking Manager
- Do the work: Import/merge the data
- QA the data
- Beverage of choice & on to the next task
The intent is to begin this effort in earnest in early 2017.
Goals
The goal of this effort is to radically improve the quality of Bellingham, WA address information in OpenStreetMap.
Schedule
- Planning: The planning is underway.
- Continuing Training: Training on how to import using JOSM will start in early 2017
- Import: This import will take place over a number of months. The City of Bellingham is updating their building outlines. Updates will be available monthly. From the monthly updates, the new outlines will be used to created suitable .osm files for import.
- QA: post-import
Import Data
Background
Data source site: City of Bellingham GIS Data Data license: Don Burdick confirmed that the data may be used in OSM. See the Contributor page for more details.
Data Files
Bellingham provides its information in various forms, including shapefile and FileGDB in Washington State Plane North. The following datasets are being used:
- https://https://www.cob.org/data/gis/FGDB_Files/COB_Land_public.gdb.zip
- https://www.cob.org/data/gis/FGDB_Files/COB_Structures.gdb.zip
- https://www.cob.org/data/gis/FGDB_Files/COB_Transportation.gdb.zip
- https://www.cob.org/data/gis/FGDB_Files/COB_Planning.gdb.zip
Import Type
This is a community building based effort.
There are currently no plans to script or automate this import.
Data Preparation
Tagging Plans
No source tags will be added.
Changeset Tag
Tag in import-related changesets:
import=Bellingham, WA GIS
Data Transformation
The source files will need to be converted to OSM XML. Shpfiles will be processed in Postgresql to expand street addresses. Two slighly different approaches are being proposed. A PostGIS approach github and a Python approach [1].
We have used Paul Norman's org2osm to convert from Postgresql to an OSM XML for this. The translation script is:
def filterTags(attrs):
if not attrs:
return
tags = {}
tags['building'] = 'yes'
if 'ADDR_NUM' in attrs and attrs['ADDR_NUM'] != '':
tags['addr:housenumber'] = attrs['ADDR_NUM']
if 'UNIT' in attrs and attrs['UNIT'] != '':
tags['addr:unit'] = attrs['UNIT']
if 'STREET' in attrs and attrs['STREET'] != '':
tags['addr:street'] = attrs['STREET']
if 'CITY' in attrs and attrs['CITY'] != '':
tags['addr:city'] = attrs['CITY']
if 'ZIP' in attrs and attrs['ZIP'] != '':
tags['addr:postcode'] = attrs['ZIP']
if 'BLDGTYPE' in attrs and attrs['BLDGTYPE'] != '':
if attrs['BLDGTYPE'] == 'CABIN':
tags['building'] = 'yes'
if attrs['BLDGTYPE'] == 'DUPLX':
tags['building'] = 'house'
if attrs['BLDGTYPE'] == 'HOUSE':
tags['building'] = 'house'
if attrs['BLDGTYPE'] == 'MOBIL':
tags['building'] = 'mobile_home'
if 'FLOORS' in attrs and attrs['FLOORS'] != '':
tags['building:levels'] = attrs['FLOORS']
if 'NAME' in attrs and attrs['NAME'] != '':
name = attrs['NAME'].title() # Probably shouldn't titlecase everything
if name[-3:] == ' Es': name = name[:-3] + ' Elementary School'
if name[-3:] == ' Ms': name = name[:-3] + ' Middle School'
if name[-3:] == ' Hs': name = name[:-3] + ' High School'
if name[:3] == 'Bfd': name = 'BFD' + name[3:]
if name[:3] == 'Bsd': name = 'BSD' + name[3:]
if name[:2] == 'Cp': name = 'CP' + name[2:]
if name[:2] == 'Fd': name = 'FD' + name[2:]
if name[:3] == 'Us ': name = 'US ' + name[3:]
if name[:3] == 'Wsu': name = 'WSU' + name[3:]
if name[:3] == 'Wta': name = 'WTA' + name[3:]
if name[:3] == 'Wwu': name = 'WWU' + name[3:]
if name[:4] == 'Ymca': name = 'YMCA' + name[4:]
# if name ends with a 2-letter WWU bldg appreviation, fix that w/ a regex?
tags['name'] = name
if 'YRBUILT' in attrs and attrs['YRBUILT'] != '':
tags['start_date'] = attrs['YRBUILT']
return tags
Data Transformation Results
Output OSM XML files can be reviewed here: https://github.com/almccon/bellingham-wa-buildings/tree/master/osm or here: https://github.com/cliffordsnow/Bellingham-Bldg-Addr/tree/master/osmcontaining all Bellingham's voting district files. Please do not import the data.
Data Merge Workflow
Team Approach
The work for this effort will be divided up into sections, with each section constituting a voting district. Voting district data have been loaded into US Tasking Manager.
References
Using the JSOM, each volunteer will begin to work through the data for that district.
- Local knowledge
- Bing aerial layer
- Existing OSM data
- Address data import
Workflow
- Click on the task tab above and claim a task on the map to the right by clicking on an area you'd like to work in
- Click the "JOSM" button, this will open the area in JOSM and load up existing OSM data
- Click the ".osm" button, this will open new address data in a separate layer
- Select the new addresses layer and validate the layer, fix all issues (more in Tips & Tricks below)
- Copy all geometry from addresses layer and paste it into the existing OSM data layer.
- Run validation on the existing OSM data layer that now contains new data, resolve all issues emanating from collisions between existing and new data (see merge rules).
- Do a sanity check on data: Do addresses correspond to adjacent roads? (more on imagery below)
- Merge with existing address data. Use the JOSM "m" key to merge nodes. Check for addresses in POI and merge to new address data.
- To merge with a building outline, use the Replace Geometry (Ctrl+Sift+G) by selection the existing building outline and the new address node. Note: Only merge building outline with an address node if the building outline contains just one address node.
- Upload data to OSM
- Go back to the Tasking Manager and mark the task as done (but not as validated)
Importer Quality Checks
The accepting tasks from the Tasking Manager should fix the following issues when importing addresses to help insure a high quality import:
- Street addresses match nearby street name. If the addr:street does not match the street, verify the OSM tag against the latest TIGER road overlay. If there is no match the district should be skipped. Notify Clifford Snow of discrepancy. Additionally, add a note asking for a verification of street names.
- Look for duplicate addresses. Check POI for address tags. Often POI's are added with incomplete addresses. Merge the imported address with the POI.
- If a single address is inside of a building outline, merge the two. Leave multiple addresses inside building outlines as individual nodes.
- Attempt to clean up misaligned roads by using the Bing background image as well as the "New & Misaligned TIGER Roads (TIGER 2013)"
Conflation Tools
Since the JOSM Conflation plugin is still broken, users will manually conflate address points
Known Dataset Conflation Issues
Local knowledge
Issue: this is a great opportunity to infuse a lot of richness into Bellingham's OSM data.
Approach: volunteers will be asked to add information they know along the way. For example: if you know the corner address for a building is a coffeeshop, add that information while you're buzzing about in the import.
Dedicated Import Account
Users will be expected to obtain a unique import account, for example, user Foo might pick Foo_Import.
QA
Validation
- Pre-import training
- Use of validation tools in the Tasking Manager process
- Group activities and IRC for question answering during the import.