Automated edits/JurgenG/European municipality websites

From OpenStreetMap Wiki
Jump to navigation Jump to search

edit: will not run script - too much complexity and uncertainty. Will share CSV files with results with community so local teams can validate and implement.

Main idea

For a different project, I'm trying to get an overview of all official municipal websites in Europe. I thought it would make sense to add the newly found website to OSM (and fix broken ones).

To prevent breaking things (again), this will happen after discussing my code with experienced OSM-devs.

Here is the current code: https://gitlab.com/jurgeng/osm-city-updater/

Look at my profile to contact me or use the GIT-flow on the GitLab repo.

Why is this relevant/important?

Often, municipalities have an official website containing essential information about this municipality. While using OSM as source, I found:

  • often the website was not in the metadata of the municipality
  • occasionally, the website stated in OSM was dead
  • regularly the website redirected to a new URL, so the one on record was not the canonical site

At first, I started uploading websites by hand - one by one. But this was a slow and cumbersome process.

How to determine the website?

There's a clear plan with an order of priority:

  1. Check if there is already a "website" field on OSM for this municipality
    1. if found: do HTTP(S) call to this url
      1. if it returns HTTP200, it's considered the correct site and nothing happens
      2. if it returns HTTP30x, pick up the redirect URL and check if this one resolves (if not a HTTP200, continue with redirect URL) ⇒ assumed to be correct → no revision needed
      3. if it returns something else (40x or 50x) → start search for other possible site
    2. if "website" doesn't exist → start search for other possible site
  2. Check if there is a website known on the Wikipedia entry for this location (perform same routine as for 1)
    1. If a Wikipedia URL is found (HTTP 200), assume this one to be the canonical one → add this URL to the OSM metadata for this municipality ⇒ assumed to be correct → no revision needed
    2. If no valid Wikipedia URL is found, continue search for other possible domain
  3. Based on common domains already found in OSM dataset, potential patterns get deduced (will happen on a per country basis)
    1. (TODO) Certain domains (e.g. TLD subdomain like .gov.uk) will be assumed reliable when found
    2. Certain domains (e.g. cityname.tld or cityname.net) will be uncertain
      1. Some are manually validated prior to submission (visit the site, see that it is the official website) → add this URL to the OSM metadata for this municipality ⇒ assumed to be correct → no revision needed
      2. Some are not manually validated prior to submission (insufficient knowledge of the language) → add this URL to the OSM metadata for this municipality ⇒ uncertain to be correct → revision required

What could possibly go wrong (or did go wrong already)

  • Did go wrong: A first iteration of the script committed website information to municipalities in Belgium, but forgot to also include the existing metadata. The consequence was that all data (except for the new website info) was removed from these municipalities. The mistake in the data got fixed by M!dgard. The code too, is fixed by now.
  • Could go wrong: Certain brands could own a domain name instead of the city (e.g. chimay.be or spa.be)
    • This is why manual validation of domains/sites is important
  • Could go wrong: Certain domains could be squatted
    • idem: manual validation

Consultation prior to execution

This script was created - not knowing about the policies about Automated Edits. This was brought under my attention here:

https://www.openstreetmap.org/changeset/162205669#map=13/50.68754/3.29006

Timing and planning

The idea is to perform this script once for every country - on a basis of available time and post-approval by community

Feel free to fork the code and pick this up for your own country.

Countries done

  • Belgium: 2025-02-06 (thanks to M!dgard for fixing my screw-up and retaining the added data)
  • ...

Licensing

As this is just indexing of factually existing websites (and not importing data from a pre-existing source), there is no licensing discussion about the legality of the content.

Sources and formats

Source for cities to be indexed

URL templates to be tried

If no website is found on OSM or Wikipedia, a few additional tries can be tapped into. These will be generated based upon the city slug (= transliteration from native city name into valid website characters)

Belgium

  • https://{city_slug}.be (requires validation - manually validated)

Netherlands

  • https://{city_slug}.nl (requires validation)

France

  • https://{city_slug}.fr (requires validation)
  • https://mairie-{city_slug}.fr (requires validation)
  • https://ville{city_slug}.fr (requires validation)
  • https://ville-{city_slug}.fr (requires validation)
  • https://agglo{city_slug}.fr (requires validation)
Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=Automated_edits/JurgenG/European_municipality_websites&oldid=2820554"