Import/Catalogue/PSMA Admin Boundaries

PSMA Admin Boundaries was an import of the PSMA Suburbs-Localities and Local Government Areas datasets, which are the administrative boundaries covering Australia. The import was carried out 27 September to 4 October 2020.

Goals

The goal was to bring the coverage of Suburbs, Localities and local government area boundaries in Australia to close to 100%. Prior to the import many of these features were either not in OSM or only existed as points.

The import was extensively discussed within the local community and the community is pro-import, see https://lists.openstreetmap.org/pipermail/talk-au/2018-October/thread.html

Schedule

Completed.

Import Data

Background

Data source site: https://data.gov.au/dataset/ds-dga-bdcf5b09-89bc-47ec-9281-6b8e9ee147aa/details
Data license: https://creativecommons.org/licenses/by/4.0/
Type of license (if applicable): CC BY 4.0
Link to permission (if required): File:Department_of_Industry_Innovation_and_Science_ODbl_permission_Administrative_Boundaries.pdf
OSM attribution (if required): https://www.openstreetmap.org/copyright
ODbL Compliance verified: yes

OSM Data Files

• https://github.com/FrakGart/psma-admin-bdy-2020-08 (simplified geometries, currently AUGUST 2020 data)

(see Data Preparation for documentation on how these were created)

Import Type

One-time import, with potential of follow-up semi-manual imports to account for changes in each quarterly upstream release.

Data Preparation

Audit of existing boundaries

Before the import an audit was made of the existing adminisrative boundaries already in OpenStreetMap (the Overpass queries below will now return the post-import number of boundaries).

Import by State

	LGA	Suburb/Localities
1. NSW	Done. Comparison of existing boundaries here	Complete, major differences are due to the use of pre-existing rivers or coastline. Smaller differences could be manually edited if needed. Comparison of existing boundaries here
2. VIC	79 LGAs in OSM. Re-import when the suburbs get re-imported.	1616 existing in OSM. Suburb boundaries appear to be an old import of the ABS2011 data when there were only 1547 suburbs in VIC, there are now 2973 suburbs. Do a re-import as manual repair will take months and be not better.
3. QLD	2 existing, to be replaced	108 existing, to be replaced
4. SA	Complete, there are differences between the PSMA and data.sa.gov.au datasets which will need to be resolved. Comparison of existing boundaries here	Complete, some need manual editing Comparison of existing boundaries here
5. WA	14 existing, to be replaced	32 existing, to be replaced
6. TAS	29 existing, need to be checked for tagging/shape.	775 existing, need to be checked for tagging/shape.
7. NT	12 existing, to be replaced	2 existing, to be replaced
8. ACT	N/A	Done.
9. OT	0 existing, to be imported	none existing, to be imported

Tagging Plans

LGA

• type=boundary
• boundary=administrative
• admin_level=6
• place=municipality
• name=titleCase(LGA_NAME)
• short_name=titleCase(ABB_NAME)
• ref:psma:lga_pid=LGA persistent identifier

Suburbs/Localities

• type=boundary
• boundary=administrative
• admin_level=10
• name=titleCase(NAME)
• ref:psma:loc_pid=Locality persistent identifier

Changeset Tags

• type=import
• source=PSMA_Admin_Boundaries
• source:date=2020-08
• url=https://wiki.openstreetmap.org/wiki/Import/Catalogue/PSMA_Admin_Boundaries

Data Transformation

https://github.com/andrewharvey/psma-admin-bdy2osm documents the work flow to get the data into shape for an OSM import. The transformations it (and QGIS) makes are summarised as:

1. Download upstream data and process to shapefiles using https://github.com/andrewharvey/psma-admin-bdys-data.
2. Preprocess shapefiles using QGIS (This is not claimed to be the best way of doing this but rather describes what was done):
   1. Use the GDAL buffer (set to 0.0) to fix the topology of the Suburbs - Localities and Local Government Areas shapefiles.
   2. Project the two layers into EPSG:4462 so they can be processed in terms of metres.
   3. Merge layers together.
   4. Apply v.clean: snap set to 0.3m, minarea 1.0m^2, break and recommended followup.
   5. Check and fix topology using GDAL buffer if required.
   6. Apply v.generalize with threshold=1.0m.
   7. Check and fix topology using GDAL buffer if required.
   8. Split layer back into localities and LGA.
   9. Dissolve areas using LGA_PID and LOC_PID.
   10. Check and fix topology using GDAL buffer if required.
   11. Project the two layers into EPSG:4326.
3. Process shapefiles into OSM format using https://github.com/andrewharvey/psma-admin-bdy2osm:
   1. Convert to GeoJSON using geojson.sh (script modified to bring along LOC_PID amd LGA_PID).
   2. Apply OSM tags to GeoJSON using geojson.sh (sl.js and lga.js modified to add ref:psma:lga_pid=* and ref:psma:loc_pid=*).
   3. Convert to TopoJSON using topojson.sh.
   4. Convert to OSM XML using osm.sh (script modified to read the pre-simplified TopoJSON as we did the simplification in QGIS).
   5. Manual fix the geometry errors (there are two crossing boundaries in the NT) and add source=* and source:geometry=* tags using JOSM. Drop unincorporated local government areas.
   6. Sort the element in the osm files using osmium sort <infile>.osm -o <outfile>.osm
   7. Renumber the elements so there are no overlapping ids using osmium renumber -s -1,-2000000,-3000000 <infile>.osm -o <outfile>.osm. This is required because the upload scripts assume that ids are unique to all OSM elements types in the file.

Data Transformation Results

• https://github.com/FrakGart/psma-admin-bdy-2020-08

Data Merge Workflow

Import Accounts

Task	Import Account
Queensland	PSMA_Admin_import_QLD (on osm, edits, contrib, heatmap, chngset com.)
Victoria	PSMA_Admin_import_VIC (on osm, edits, contrib, heatmap, chngset com.)
Western Australia	PSMA_Admin_import_WA (on osm, edits, contrib, heatmap, chngset com.)
Tasmania	RoundCircle (on osm, edits, contrib, heatmap, chngset com.)
Northern Territory	PSMA_Admin_import_NT (on osm, edits, contrib, heatmap, chngset com.)
Other Territories	PSMA_Admin_import_OT (on osm, edits, contrib, heatmap, chngset com.)

References

List all factors that will be evaluated in the import.

Workflow

Steps followed during import are listed below.

Pre-import

1. Split boundary with adjacent state/territory where the level 6 and 10 boundaries will meet (example 91757559 ).
2. Take stock of existing level 6 and 10 boundaries.
3. Tag existing relations with the most appropriate ref:psma:loc_pid=* or ref:psma:lga_pid=* (example 91920316 ).
4. Remove ways from existing relations and then delete the ways that are not used for anything else:
   1. Find the ways that will be orphaned when they are removed from either the level 6 or 10 relations using Overpass
   2. Tag ways with a fixme that can be found later (example 91690444 ).
   3. Remove ways from 6 and 10 relations (example 91733980 ).
   4. Download possible orphaned ways using Overpass
   5. Run JOSM validator and delete unused ways (example 91735130 ).
5. Existing relations can be found again using Overpass
6. Replace ways in the import file with the coresponding existing OSM way

• Choose a state, as the import will be done one state at a time
• Use existing state boundary
• Done using a dedicated imports account

Conflation

Conflation involves replacing the outer boundaries of the PSMA dataset with the existing OSM state border. This will be done manually before uploading.

Post Import

Most if not all localities in Australia are already mapped as nodes. We should take this into account as we blend the PSMA data with OSM data.

1. Add a place=* node if one is missing for the locality/suburb.
2. Make sure the boundaries have a place=* tag, have it match what is on the nodes.
3. General spell and capitalisation check on names
4. Add the nodes into the relation, and set the role to "label" (example: https://www.openstreetmap.org/relation/6070036)

And for general improvement to OSM, the following are highly recommended

• Reconsider the place=* value (Is it really a village or a suburb? Is it really a city or is it a town?)
• Reconsider the placement of the node (For rural places it should be roughly in the centre, for populated areas put it near a central town hall, square or similar. A possible option is to use this methodology https://github.com/iag-geo/psma-locality-centroids to place the node near the centre of habitation.

QA

Before upload the proposed boundaries will be checked using the JOSM validator. Sucess of upload can be checked by downloading the new boundaries and comparing them to the proposed upload.

See also

The email to the Imports mailing list was sent on 2018-10-22 and can be found in the archives of the mailing list at https://lists.openstreetmap.org/pipermail/imports/2018-October/005760.html

The discussion within the Australian community was ongoing on the talk-au mailing list at https://lists.openstreetmap.org/pipermail/talk-au/2018-August/thread.html, https://lists.openstreetmap.org/pipermail/talk-au/2018-October/thread.html, and https://lists.openstreetmap.org/pipermail/talk-au/2020-September/014009.html