Overpass API/Augmented Diffs

Overpass API · Language reference · Language guide · Technical terms · Areas · Query examples · Sparse Editing · Permanent ID · FAQ · more · Web site
Servers status · Versions · Development · Technical design · Installation · XAPI compatibility layer · Public transport sketch lines · Applications · Source code and issues

Overpass turbo · Wizard · Overpass turbo shortcuts · MapCSS stylesheets · Export to GeoJSON · more · Development · Source code and issues · Web site

Overpass Ultra · Examples · Overpass Ultra extensions · MapLibre stylesheets · URL Params · more · Source code and issues · Web site

The augmented diffs extend the ordinary minute diffs. They make it possible to keep even specialized databases up to date minutely. They are based on an idea by Matt Amos, refreshed by Frederik in a talk in German.

Augmented diff API endpoint: http://overpass-api.de/api/augmented_diff (stable endpoint with long term support)

Overview

The augmented diff offers comparison of the result of the same query in two different times. The final result contains all the data of nodes/ways/relations with difference between the starting time and the ending time. (It does not take notice of any intermediate change.) For example query and results see: http://overpass-turbo.eu/s/dMf

Each returning element is noted with one of the three actions:

• delete: when the element does not match the query at the ending time.
• modify: when the element matches the query at both times but it has changed in other aspects.
• create: when the element has not matched the query at the starting time.
• (If an element matches the query both at starting and at the ending time but it is exactly the same at both time (not taking notice of any intermediate change) so this element would not appear among the results.)

The augmented diffs contain three kinds of data of an element:

• The new data that comprises the usual minute diffs.
• The old data that is either replaced by new data or explicitly deleted
• Meta data for deleted objects

In addition, the ways and relations are enhanced by their geometry.

This allows in particular the following two applications

• Proper minute updates on datasets restricted by a bounding box
• A wayback mechanism: You can start with a planet file or other database and walk back instead of forward through time

While the use of the former is immediately clear, the second needs some explaining words: A couple of examples from the talks above like the counting application need to know what is deleted to decrease the counter.

Contained data

Each diff is structured in three sections, one for each type of element. Within a section, the elements are ordered by id. Each element only appears its state at the beginning and the state at the end of the minute. Shorter lived versions are omitted.

Every element is considered as having changed when its meta date has been updated or its tags have changed. In addition, nodes are considered to have changed when their coordinates have changed. Ways are considered as changed when their members have changed or one of its members has changed its coordinate. Relations are considered as changed when their members have changed or one of its way members has changed by its members or coordinates or one of the node members has changed its coordinate. By contrast, changes on members of type relation don't propagate to their parent relations.

In the preamble, the file contains the end date of the diff.

<?xml version="1.0" encoding="UTF-8"?>
<osm version="0.6" generator="Overpass API">
<note>... </note>
<meta osm_base="2012-09-20T20\:24\:02Z"/>

  <!-- Elements are ordered as: nodes first, then ways, then relations.
       Within each class of elements they are ordered by id -->

</osm>

Actions

Each element is wrapped in an action of type delete, modify, or create, depending on whether the element matches the query at the beginning and/or the ending time.

delete

An object is noted as deleted if the element does not match the query at the ending time. Because this produces a new version, there exists also a "new" section. This contains an empty OSM element with a "visible" marker. If the element is deleted from the OSM database then "visible=false". If the element is not delete from the OSM database only the element does not match the query at the ending time then "visible=true".

<action type="delete" ...>
  <old>
    <!-- OSM element -->
  </old>
  <new>
    <!-- OSM element -->
  </new>
</action>

modify

An object is noted as modified if the element matches the query at both times but the element has differences between the starting time and the ending time. Here, the new version is the proper new version of the element at the ending time.

• Please note that for ways and relations old and new version could have the same version number. This happens when the element itself is unchanged but a member element has changed and thus also the geometry of the element itself.
• Please also note that Augmented Diff does not care about any intermediate changes only cares about differences between the starting and ending times.

<action type="modify" ...>
  <old>
    <!-- OSM element -->
  </old>
  <new>
    <!-- OSM element -->
  </new>
</action>

create

An object is noted as created if the element does not match the query at the beginning time but it matches the query at the ending time. This object only contains one version, the version of the element at the ending time.

• Please note that being noted as created does not mean that the element was created after the beginning time. It means only that it does not match the query at the beginning time.
•  Also be aware that the element could match the query at a time before the beginning time. Create means only that the element does not match the query at the very minute of the beginning time.

<action type="create" ...>
  <!-- OSM element -->
</action>

Elements

All way and relation elements contain a "bounds" element. This elements tells you the enclosing bounding box.

All nd elements inside of way elements contain additional lat and lon attributes:

 <way id="..." ...>
   <bounds ... />
   <tag ... />
   ...
   <nd ref="..." lat="..." lon="..." />
   <nd ref="..." lat="..." lon="..." />
   <nd ref="..." lat="..." lon="..." />
   ...
</way>

All members of type node of relations also contain additional "lat" and "lon" attributes:

 <rel id="..." ...>
   <bounds ... />
   <tag ... />
   ...
   <member type="node" role="" ref="..." lat="..." lon="..." />
   <member type="node" role="" ref="..." lat="..." lon="..." />
   <member type="node" role="" ref="..." lat="..." lon="..." />
   ...
</rel>

Members of type way repeat their geometry in a sequence of "nd" subelements:

 <rel id="..." ...>
   <bounds ... />
   <tag ... />
   ...
   <member type="way" role="" ref="...">
     <nd lat="..." lon="..."/>
     <nd lat="..." lon="..."/>
     <nd lat="..." lon="..."/>
   </member>
   ...
</rel>

Time slices and numbering

The augmented diffs are numbered with consecutive numbers. The zero point is the license change, i.e.

http://overpass-api.de/api/augmented_diff?id=0&debug=yes

will tell you that it is going to fetch the difference between "2012-09-12T06:55:00Z" and "2012-09-12T06:56:00Z". From that date on, the counter is increased once per minute.

You can get the last available augmented diff via the API call

http://overpass-api.de/api/augmented_diff_status

Deriving augmented diff ("adiff") identifier from a timestamp

The following bash command computes the augmented diff id that contains the "2017-01-16T15:35:17Z" timestamp.

   echo $((( $( date --date="2017-01-16T15:35:17Z" +%s ) + 59)/60 - 22457216))

NOTE:

An augmented diff id is created at the 00 second of each minute, and it contain new changes up-to that time. For example, the above augmented diff id 2285800 includes changes made during the interval "2017-01-16T15:35:01Z" to "2017-01-16T15:36:00Z".

The bash formula here takes the UTC time (2017-01-16T15:36:16Z), converts it to seconds, adds 59 seconds to align with the interval's end minute, and divides that by 60 to get minutes since the start of the UNIX epoch in 1970. It then subtracts the delay to the start of the OSM change epoch (Wed, 12 Sep 2012 06:56:00 UTC, which is, 22457216 minutes since start of UNIX epoch) to get the number of minutes since Wed, 12 Sep 2012 at 06:56:00 UTC, which is the augmented diff identifier.

The augmented_diff API call

On overpass-api.de exists an API that allows to filter the Augmented Diffs already on the server. The base URL is

 http://overpass-api.de/api/augmented_diff?

It accepts the following two arguments:

• id= is mandatory and is the id of the Augmented Diff to process
• bbox= is optional and is a min_lon,min_lat,max_lon,max_lat limited bounding box of the area of interest

Applications

Software and Services using Augmented Diffs:

• osmconvert (since 0.7E): The program can now read Augmented Diffs and convert them to .osc. It also can process them and update an existing .osm or .pbf. All this is highly experimental, you should expect one or two bugs... :-)
• Osm-watch: osm-watch is an OSM contributions almost real-time monitoring tool with filters based alerts that you can created and receive by RSS or email.
• achavi: Augmented Change Viewer - visualizes updates to OpenStreetMap
• Show Me The Way on OSMLab
• osmdiff: A Python SDK for working with OsmChange and Augmented Diff files. Able to retrieve latest / specific diffs from the Overpass / OSM API.
• osm-revert: A web-based changeset reverter with automatic conflict resolution and no size limits.
• OSMCha: a web-based changeset viewer and review tool which uses Augmented Diffs internally