Ottawa/Import/Plan
Background
Essentially Statistics Canada are interested in adding details to non-residential buildings in Ottawa / Gatineau. The sort of details they are after in my mind will require a physical mapper to look at the building. This is traditional OpenStreetMap mapping, nothing else. The intent is not to wave a magic wand and pour all sorts of data into OpenStreetMap.
However they would like some reassurance of completeness. Have we mapped all the buildings?
The buildings they are interested in probably number around 5,000 and many / most are in the map already although they do not have all the tags Stats Canada would like. I can't remember if the colour of the door was one but they are of that type.
After a conversation between Stats Canada and The City of Ottawa with some other people involved a file containing all the buildings in Ottawa was identified.
The City of Ottawa agreed to make this available. By the time it was made available the City of Ottawa had changed its Open Data license, the process took three years by the way to one which is for all practical purposes the same as the Federal Government's Open Data portal. My understanding is the that file is currently available on one of the City of Ottawa's servers but I may be wrong. The original intention was it would be made available on the Treasury Board's Open Data web site which is normally considered to be compatible with OpenStreetMap's .odbl license. I'm not quite certain if it has made its way to the Federal Government's Open Data Portal yet but that was the intention and I don't think that has changed.
There will always be different points of view about whether or not a particular Open Data license is compatible with OpenStreetMap but I think in this case considerable effort and research was made to ensure the licensing was correct. I distinctly recall murmurings of frustration being heard at Stats Canada about licensing on more than one occasion.
There have been a number of meetings / discussions, four or five I think, with the local mappers that were open to all. A group physically meets up at the Happy Goat in Ottawa from time to time. I don't think AJ Ashton nor Stewart Downs were amongst them but they were invited. The local group is hosting a tile whatever to cover Ottawa in support of this project. I think I was at a meeting at a Starbucks where there was a discussion by mappers about the building import and the best process to use to bring in the building outlines, but at all the meetings to my knowledge there was mention that in order to avoid the drawing odd shaped buildings in iD and to ensure that all the buildings were known about there would be use of Open Data in support of the project.
There has been some discussion in talk-ca.
Statistics Canada have been talking to people in Brussels at SOTM and also at the American OSM SOTM.
Within the wider OSM community there has been some discussion and much interest.
This is the first time Statistics Canada has been involved with something of this nature and they are still learning. They are trying to work with the local mapping community but internally they also have some target dates they would like to meet.
The import was done by the local community.
I think what should have happened is the intention of the import should have been raised as a courtesy with the import discussion forum but having seen their input about CANVEC data being imported they seem to have a less than helpful reputation which is unfortunate.
Stats Canada was hoping to announce the project today and I think they now have done so.
http://www.statcan.gc.ca/eng/crowdsourcing
So it may have been tempting to add a few buildings before they started.
It should have been done using import OSM user ids.
It should have been done carefully only importing where there were no existing buildings. This takes a lot longer and for an import of this size the temptation would probably be to import the more rural areas where fewer buildings are mapped and be much more careful in the city centre.
So currently the Statistics Canada project is now live and they are asking people to add data or rather tags to existing buildings and there are quite a few existing buildings on the map in Ottawa that can be worked on already. This is slightly different than using iD to draw buildings which has led to some rather strange shaped buildings on the map. I'm hoping OpenStreetMap will gain more than a few new mappers with this initiative and for that reason alone it is well worth supporting.
By the way does anyone have any input about how local local is? It is impossible to contact and gain consensus of everyone who has ever mapped in Ottawa or Gatineau. A reasonable attempt was made to reach out to local mappers.
Permission
date: Wed, Oct 19, 2016 at 4:14 PM
to: Robert Giggey <Robert.Giggey at ottawa.ca>,
Stephen M Perkins <stephen.perkins at ottawa.ca>,
bill.harper at ottawa.ca
Hi as Stephen is out of the office I will contact Bill as well. As per documentation reasons. Some people in the OpenStreetMap community would like it documented that the City of Ottawa has handed over building footprints and address data to Stats Can for use in OpenStreetMap and the Stats Can project. I know I got your approval back in 2015, but they dont want to rely on the word of the federal government.
--------------------------------------------------------------------------------------------------------------------------------------------------------
date: Thu, Oct 20, 2016 at 9:11 AM
to: James <xxxxxxxxxx@gmail.com>
cc: "Giggey, Robert" <Robert.Giggey at ottawa.ca>,
"Perkins, Stephen M" <Stephen.Perkins at ottawa.ca>,
"bjenk.ellefsen@canada.ca" <bjenk.ellefsen at canada.ca>,
"Flowers, Andrea" <Andrea.Flowers at ottawa.ca>
James,
I have a data licence agreement dated July 20, 2016 that transferred our building footprint file in the urban area to Stats Canada. This information went to Bjenk Ellefsen (I have ccd Bjenk on this note) and was intended for a ”Crowdsourcing data collection project”. Having said that …if Stats Can is using a framework for data input and they are opening that to the Public then I don’t see an issue from our perspective.
Hope this helps
Bill
W.A. (Bill) Harper
City Surveyor
Explicit permission
Follow up email:
From: Giggey, Robert [mailto:xxxxxxx@ottawa.ca] Sent: January-24-17 10:21 AM To: Ellefsen, Bjenk (STATCAN) <xxxxxx@canada.ca> Subject: RE: Project with StatCan and Open Data Hi, I was going to reply today. I can’t license it to them under a separate license. We’ve gotten approval to share through the open data license. I don’t believe that keeps them from using it in OSM. Hopefully that will be enough for them. Robert Giggey Program Manager | Web Services ServiceOttawa
from: Giggey, Robert <xxxxxx@ottawa.ca> via ottawa4.onmicrosoft.com to: James <xxxxx@gmail.com> cc: "Ellefsen, Bjenk (STATCAN)" <xxxx@canada.ca>, "Perkins, Stephen M" <xxxxxxx@ottawa.ca>, "Harper, Bill" <xxxxxxxxx@ottawa.ca> date: Tue, Jan 24, 2017 at 5:27 PM subject: RE: Stats Can project Hi James, while I can’t publish our data under the odbl I can say that it appears to be in-line with our ODL, and I can’t see any issues with it being published to OpenStreetMap. Robert Giggey Program Manager | Web Services ServiceOttawa twitter | linkedin | yammer
Current Status of Project
Phase 1 (In progress): 19th of October discussion of import started on Talk-ca and import mailing list
Phase 2 (Not started): Training and talking to the ones participating in the import on best practices/what to avoid
Phase 3 (Not started): Import building outlines and address data via tasking manager
Points of Contact about project
Statistics Canada
Bjenk Ellefsen [bjenk.ellefsen at canada.ca]
City of Ottawa
Stephen Perkins [stephen.perkins at ottawa.ca] Bill Harper [bill.harper at ottawa.ca]
Licence
The city of Ottawa is publishing their data set under the City of Ottawa Open Data Licence Version 2.0. This license is derived from the Open Government Licence 2.0. Modifications to the license are to remove the federal aspect of Canada's version of the license and to apply it to a municipal level.
The OSMF Licensing Working Group determined in their meeting on 2017-03-02 that data under the Ottawa Open Data, Licence Version 2.0 (Ottawa ODL 2.0) can be included in the OpenStreetMap dataset and distributed on ODbL 1.0 terms. Quoting from the draft minutes (draft link may not be generally readable):
The LWG has determined [(1)] that the attribution requirements of the Ottawa ODL 2.0 can be met by adding the required text to the wiki contributor page and corresponding changeset source attribute values, and that there is no downstream attribution requirement, [(2)] that we are not using "Personal Information" as defined in the licence and referenced legislation, and that so licensed material can be included in the OpenStreetMap dataset and distributed on ODbL 1.0 terms.
— OSMF LWG draft minutes, 2017-03-02
Note: The LWG's acceptance of the Ottawa ODL 2.0 or the Canada Open Data Licence 2.0 does not mean that near-identical licences are also acceptable for OSM import. The draft minute goes on to say:
In the past the local variants of the OGL Canada have varied widely and have in some cases included additional terms that have made them incompatible with the ODbL and in some instances non-open. For this reason we are not making a blanket statement on other such localised versions of the OGL at this point in time and will continue to review them on a case by case base.
For example, if the fictional City of Rotonto took the exact text of the Ottawa ODL 2.0 and merely replaced instances of “Ottawa” with “Rotonto”, the above minute indicates that the Rotonto ODL would still need LWG approval.
Import Plan Outline
Goals
- Import the city of Ottawa's excellent building footprints data (missing and update really bad geometry) to help Ottawa mappers create a more effective map.
- Get highly accurate, city-wide address data.
Schedule
- 20 October 2016: discussion period of two weeks, starting when posted to imports@ list and talk-ca@ list
- if positive result of discussion: import after end of discussion period
Import Data
The data used for this import is are two datasets cleaned up by the OSMCanada team one containing building outlines and the other address points.
Original Dataset (addresses): Address Points
Original Dataset (building footprints): Building footprints
Data source site: Both will be served through a data service(as clean up needed to be done. The service will serve the data in MBTiles Vector Tile 2.0 format, thus you must provide a z,x,y to obtain data.
- Address (Points) https://data.osmcanada.ca/{z}/{x}/{y}/ottawa-address.osm
- Building Footprints (Polygons) https://data.osmcanada.ca/{z}/{x}/{y}/ottawa-buildings.osm
Data license: See above with also written permission we also have permission from the city in a collaborative effort with Statistics Canada
OSM Data Files
The data will be served by https://www.data.osmcanada.ca/ as JOSM xml files via a link in the tasking manager. See data source site for more information.
Import Type
This is a one-time import, will be prepared via script and tasked via a tasking manager
Data preparation
Data will be cleaned of any duplicate nodes as well as way simplification. It will then be tasked via the OSMCanada Tasking Manager (http://tasks.osmcanada.ca)
Attribution tags
source=City of Ottawa
Used Tags
Address Shapefile fields
All addresses contains these tags.
| Shapefile Attribute | OSM Tag | Description |
|---|---|---|
| addrnum | addr:housenumber=* | Civic address |
| unitid | addr:unit=* | Unit number/Letter |
| fullname | addr:street=* | Street name with strert type abreviations(ST, PKW, RD,etc) and street direction. Needs to be preprocessed to comply with OSM naming conventions |
Buildings
Building footprint is the only thing being pulled in. No tags will be translated. If bad geometry is present, it will be updated with Ottawa, else it will be left alone. The import process can separate existing buildings in osm vs new buildings and the main goal will be to add missing buildings as opposed to updating all geometries. Sheds will be excluded via a soft coded filter at the end of the URL(a.k.a is modifiable by importer) so that anything that is smaller than 300 sq feet is ignored(customizeable)
Data Workflow
Video explanation
Team Approach
It's planned to be done by a few members of OSMCanada located in Ottawa and possibly a few experienced OSM Users. The tasking manager will be used to coordinate the import. The first pass will concentrate on the buildings. Once the buildings in the tile are imported the task will be marked as complete. The second pass will use the same task, but for the addresses. Once the addresses are imported/merged to the buildings in a tile, that tile will be marked as validated. We will be working from the outskirts of Ottawa moving towards downtown(which we will have a discussion on how to proceed from there as there are a bunch of buildings already existing and conflation of data will take longer)
Risks
Risks associated with this are as follows
1. Buildings already exist in OSM
2. Address node might be in backyard instead of on the building instead of the building footprint
3. Multiple buildings are present and it is not clear what address goes with which
Mitigation
How we are going to avoid the problems related to the risks
1. We will replace geometry (if really bad), using the replace geometry function in JOSM, with the new data if we determine it is better than the current. This will keep the entire history for the current item as well as keep the same OSM id. The data service can sort out existing buildings vs new ones and there will be two separate links for import
2. Addresses will be merged down to the buildings in a second pass of the import. The first pass should concentrate on the buildings, however aligning the address nodes on top of where they are supposed to go or merged down will be highly encouraged as to avoid problems and save time in the second pass over
3. To avoid assuming, a note should be opened on OpenStreetMap i.e. https://www.openstreetmap.org/note/741666 to request a local survey to avoid errornous data
QA
Quality assurance will be done by people using the tasking manager. As well as Osmose and other QA tools to ensure proper compliance.
Issues
Footprints which don't match aerial imagery: