Montréal/Imports/Adresses ponctuelles

Montreal Adresses ponctuelles is a completed import of the City of Montreal's Adresse ponctuelle dataset.

Goals

The goal of this import is to replace existing CanVec address interpolation for the territory of the agglomeration of Montreal.

Import Data

Background

Data source sites: Adresse ponctuelle
Data license: Licence d’utilisation des données ouvertes de la Ville de Montréal (note the section "Utilisation des données par OpenStreetMap").
Type of license: CC-BY-4.0.
Link to permission: Montréal/Imports/Ville de Montréal.
OSM attribution: Contributors (see "Montréal").
ODbL Compliance verified: yes.

Import Type

This is a one-time import that will be added to the OSM database via JOSM.

Data Preparation

Tagging Plans

The imported nodes will have the following tags:

• addr:city=*
• addr:housenumber=*
• addr:street=*
• addr:province=QC

The address interpolation ways will have:

• addr:interpolation=*

The use of virtual address interpolation ways is noted on the Wiki here (ctrl + f: "virtual"). There is no consensus as to whether applying addr:interpolation=* to an address range specified using a delimiter is appropriate; see the last point of the section "More unusual cases" on the Addresses Wiki page. As a result, the first approach will be used.

Changeset Tags

Key	Value
comment	Replacement of CanVec address interpolation with the City of Montreal's Adresse ponctuelle dataset (CanVec interpolation will be deleted shortly).
import	yes
source	Ville de Montréal
source:url	https://donnees.montreal.ca/dataset/adresses-ponctuelles;https://donnees.montreal.ca/dataset/limites-administratives-agglomeration
source:date	2024
import:page	https://wiki.openstreetmap.org/wiki/Montréal/Imports/Adresses_ponctuelles
source:license	CC-BY-4.0

Data Merge Workflow

Workflow

Data preparation

1. Download the Addresse ponctuelle dataset as a geojson.
2. Open the dataset in QGIS.
3. Export the features as a geopackage in MTM 8. Export only the following fields:
   • SPECIFIQUE
   • LIEN
   • GENERIQUE
   • ADDR_DE
   • ADDR_A
   • ORIENTATION
4. Change the project's projection to match.
5. To match existing OSM street name capitalization, we need to capitalize the first letter of the values in the GENERIQUE field. To do so, open the field calculator and create a string field with a length of 0 named gen title. Paste the following expression: title( "GENERIQUE" ) . Save changes.
6. To create full street names, create a string field named addr:street_inc. Run the following expression:
   • if(regexp_match("SPECIFIQUE", '^[0-9].*e$') != 0 AND ("gen title" = 'Avenue' OR "gen title" = 'Rue'), to_string("SPECIFIQUE" || ' ' || "gen title"), if( "LIEN" IS NULL, to_string ( "gen title" || ' ' || "SPECIFIQUE" ), if( "LIEN" = 'd\'' OR "LIEN" = 'de l\'', to_string ( "gen title" || ' ' || "LIEN" || "SPECIFIQUE" ), to_string ( "gen title" || ' ' || "LIEN" || ' ' || "SPECIFIQUE" ))))
   • Then, create a string field named addr:street and run the following:
   • CASE WHEN  "ORIENTATION"  = 'N' THEN  (  "addr:street_inc"  || ' Nord' ) WHEN  "ORIENTATION"  = 'S' THEN  (  "addr:street_inc"  || ' Sud' ) WHEN  "ORIENTATION"  = 'E' THEN  (  "addr:street_inc"  || ' Est' ) WHEN  "ORIENTATION"  = 'O' THEN  (  "addr:street_inc"  || ' Ouest' ) ELSE  "addr:street_inc"   END
7. Download the Limites administratives de l'agglomération de Montréal dataset as a geojson in mtm 8. Add it to the project.
8. Save the layer from step 7 as a geopackage in mtm 8. This is for performance reasons.
9. Run the Join attributes by location tool. Join to features in the layer created in step 3. Compare it to the features from the layer added in step 8. Add only the following fields: NOM and TYPE. Make the join type one-to-one (largest overlap).
10. Create a new string field called addr:city. Run the following expression: if( "TYPE" = 'Arrondissement', 'Montréal', "NOM" ) .
11. Export the layer from step 10 as a geopackage, exporting only the following fields: addr:city, addr:street, ADDR_DE and ADDR_A.
12. Run the Add autoincrement field tool on the layer created in step 11.

Gathering the required OSM data

1. Open JOSM. Create a bounding box that covers at least the area of the agglomeration of Montreal. Then execute the following overpass search string: "addr:housenumber"=* OR "addr:interpolation"=*.
2. Save the data in osm format (to avoid repeatedly querying overpass).
3. Search for the following: ("addr:city"=* AND "addr:housenumber"=* AND "addr:street"=* AND source~.*NRCan-CanVec.* AND type:node AND tags:4 AND ways:1) OR ("addr:interpolation"=* AND source~.*NRCan-CanVec.* AND type:way AND -closed AND tags:2).
4. Invert selection.
5. Create a new layer and merge the selection into it.
6. From the layer created in step 5, search: "addr:housenumber"=* AND type:node.
7. Create a new layer and merge the selection into it. Save as a geojson.
8. From the layer created in step 5, search: "addr:housenumber"=* AND type:way closed.
9. Select way nodes. Note: the UtilsPlugin2 needs to be installed.
10. Repeat step 8, this time adding to the current selection.
11. Create a new layer and merge the selection into it. Save as a geojson.

Determining which addresses should not be imported and preparing data for JOSM

1. Import a copy of the layer created in step 7 of the section above into QGIS.
2. Import a copy of the layer created in step 11 of the section above into QGIS. When asked, import only the polygons.
3. Run the Centroids tool. The input layer is the one added in step 2 of this section.
4. Reproject the layer added in step 1 of this section to MTM 8 and save as a geopackage.
5. Reproject the layer created in step 3 of this section to MTM 8 and save as a geopackage.
6. Run the Merge vector layers tool. The input layers are those created in steps 4 and 5. The destination CRS is MTM 8. Save as a geopackage.
7. Export the layer created in step 6 as a geopackage in MTM 8. Keep only the addr:housenubmer field.
8. Run the select by location tool. Select features from the layer created from the previous step. Compare to features of the layer created in step 8 of the first section of steps. Export the selected features as a geopackage in MTM 8.
9. Run the Join attributes by nearest tool. Input layer 1 is the layer created in step 8 of this section. Input layer 2 is the layer created in step 12 of the first section of steps. Make the prefix "mtl_."
10. Select using the following expression:
    • "addr:housenumber" LIKE ('%' || "MTL_ADDR_DE" || '%') AND "addr:housenumber" LIKE ('%' || "MTL_ADDR_A" || '%')
11. The selection is the City of Montreal addresses we do not want to import, as they already exist in OSM. Export the selection as a geopackage in MTM 8.
12. Run the Join attributes by field value tool. The Input layer is the layer created in step 12 of the first section of steps. Input layer 2 is the layer created in step 11 of this section. The table fields are AUTO and mtl_AUTO respectively. Join type is one-to-many. Save the unjoinable features as a geopackage in MTM 8.
13. Run the Buffer tool. The input layer is the layer created in step 12 of this section of steps. The distance is 30 m. Save as a geopackage in MTM 8.
14. Run the Join attributes by location tool. The input layer is the one created in step 13 of this section of steps. The "by comparing to" layer is the one created in step 8 of this section of steps. Add only the "addr:housenumber" field. Keep the join type as one-to-many. Discard records which could not be joined. Save as a geopackage in MTM 8.
15. Run the Aggregate tool. The input layer is from the previous step. Group by the AUTO field. Delete all aggregates. Create a new aggregate with the following properties: source expression is the addr:housenumber field; the aggregate function is concatenate unique; the delimiter is , (note the space after the comma); the name of the new field is hn_cat_u and it's a string field. Create a new aggregate with the following properties: source expression is the AUTO field, the aggregate function is first value, the name of the new field is AGG_AUTO and it's a string field. Save as a CSV file.
16. Using the field calculator, for the layer created in step 12 of this section of steps, create a new string field titled AUTO_str. Use the following expression: to_string(  "AUTO"  ) .
17. Run the Join attributes by field value tool. The input layer is the one created in step 15 of this section of steps. The table field is AGG_AUTO. The second input layer is the one created in step 12 of this section of steps (and modified in the previous step). Its table field is AUTO_str. Make the join type one-to-many (although this shouldn't matter). Save as a CSV.
18. Select using the following expression: "hn_cat_u"  LIKE ('%' ||  "ADDR_DE"  || '%') AND  "hn_cat_u"  LIKE ('%' ||  "ADDR_A"  || '%'). Save the selection as a CSV.
19. Run the Join attributes by field value tool. Join the layer created in step 12 of this section of steps to the layer created in the previous step. The table fields are both AUTO_str. Save the unjoinable features as a geopackage in MTM 8.
20. Run the "Points displacement" tool on the most recently created layer. The minimum distance to other points is 0.6 m. The displacement distance is 1.2 m.
21. Select using the following expression: "ADDR_DE" = "ADDR_A" . Export the selection as a geojson in WGS 84.
22. Invert the selection and export it as a geopackage in MTM 8.
23. Export the most recently created layer, but omit the ADDR_DE field from the export.
24. Run the Array of translated features tool. The input layer is the one from step 22. Number of features to create: 1. The step distance on the x-axis is -0.5 m (note the negative). Save as a geopackage in MTM 8.
25. Run the same tool again, this time on the layer created at step 23. Number of features to create: 1. The step distance on the x-axis is 0.5 m. Save as a geopackage in MTM 8.
26. From the layer created in step 24, select points from instance 1 and export this selection as a geopackage in MTM 8.
27. Repeat the above step for the layer created in step 25.
28. Merge the layers created in steps 26 and 27 together. Save as a geopackage in MTM 8.
29. Run the Points to path tool. The input later is from the previous step. The path group expression is AUTO. Save as the usual geopackage.
30. Export the layer from step 26 as a geojson in WGS 84. Export only the following fields ADDR_DE, addr:city and addr:street.
31. Export the layer from step 27 as a geojson in WGS 84. Export only the following fields ADDR_A, addr:city and addr:street.
32. Join the layer from step 29 to the layer from step 22. Add only the ADDR_DE field. Make the join type one-to-one (largest overlap). Save as a geopackage in MTM 8.
33. Use the field calculator to create an integer field named addr_de_int. Use the following expression: to_int( "ADDR_DE" ) .
34. Use the field calculator to create a string field named addr:interpolation. Use the following expression: if( "addr_de_int" % 2 = 0, 'even','odd').
35. Export this layer as a geojson in WGS 84. Export only the addr:interpolation field.
36. Dissolve the layer from step 7 of the first set of steps. Save as a geojson.
37. Reproject this layer in WGS 84 and save as a geojson.

Opening the data to be imported in JOSM

1. Open the geojson files obtained from steps 21, 30, 31 and 35 of the section above in JOSM.
2. Make the layer from step 21 the active layer and select all.
3. Delete all fields except: ARRD_DE, addr:city and addr:street.
4. Change the key of ADDR_DE to addr:housenumber.
5. Make the layer from step 30 the active layer and select all.
6. Change the key of ADDR_DE to addr:housenumber.
7. Make the layer from step 31 the active layer and select all.
8. Change the key of ADDR_A to addr:housenumber.
9. Make the layer from step 35 the active layer.
10. Search for type:node.
11. Freeze as the subject in the conflation plugin.
12. Merge the layer from step 31 into the layer from step 30.
13. From the resulting layer, select all and set as the reference in the conflation plugin. Set the method to One to one. Distance to Centroid and 0.1. Check (only) Merge tags, All and Overwrite without confirmation. Generate matches. This step will take a while.
14. Conflate all matches.
15. Make the layer from step 21 the active layer. Merge it into the layer from step 35.
16. Save the most recently modified file as an osm file.

Fix: removal of unnecessary interpolation ways

Context: the Nominatim QA tool states that interpolation ways without a missing number are unnecessary.

1. On the layer created in step 28 of the third section of steps, select using the following expression:   ( to_int(  "ADDR_A"  )  -  to_int(  "ADDR_DE"  ) )   = 2.
2. Using the select by location tool, select features from the layer created in step 32 of the third section of steps. Compare features from the layer whose points were selected from in the previous step. Check "Selected features only."
3. Export the selection.
4. Create centroids for this newly created layer.
5. Switching from QGIS to JOSM, open the file to be imported as well as a geojson of the layer created in step 4 of this section of steps.
6. Add a note=* key-value pair to the centroids layer (to make selection easier).
7. Merge the layer with the note=* into the other open layer.
8. Search for note=*.
9. Join selected nodes to way.
10. Search for note=*.
11. Select adjacent ways and delete them.

How the CanVec data will be removed

Note: this will only be done a few hours after the import of the Adresses ponctuelles dataset. This is to avoid running into any changeset limit cool-downs and to allow for enough time to confirm all was properly imported.

1. In JOSM, create a bounding box at least as large as the agglomeration of Montreal.
2. Download the relevant extent using overpass (submit empty dialog).
3. Open the layer created in step 37 of the section Determining which addresses should not be imported and preparing data for JOSM.
4. Merge this into Data layer 1 (temporary).
5. On Data layer 1, select this merged way and then select All inside.
6. Search for the following (find in selection): "addr:interpolation"=* AND source~.*NRCan-CanVec.* AND type:way AND -closed AND tags:2 AND -type:relation AND -parent type:relation AND -child type:relation.
7. Select way nodes.
8. Search for the following (find in selection) untagged AND ways:1 AND -"source"=* AND -type:relation AND -parent type:relation AND -child type:relation. Delete this selection.
9. Repeat steps 5, 6 and 7.
10. Search for the following (find in selection): "addr:city"=* AND "addr:housenumber"=* AND "addr:street"=* AND source~.*NRCan-CanVec.* AND type:node AND tags:4 AND ways:1 AND -type:relation AND -parent type:relation AND -child type:relation. Delete this selection.
11. Repeat steps 5 and 6. Delete selection.
12. Remove the temporary polygon used to select.

Changeset size policy

In JOSM, upload objects in chunks of size: 9950.

The addresses will be imported before the CanVec address interpolation is removed (to make reverting the changes easier, if ever something goes wrong).

Revert plans

If ever something goes wrong, the JOSM/Plugins/Reverter plugin will be used.

List of OSM IDs created

A list of the objects that were created by this import is available here.

See also

To reach a consensus on how to represent the imported addresses for buildings with more than one address, a forum post was made here.