User:mmd/Overpass API - experimental corner

Overpass API · Language reference · Language guide · Technical terms · Areas · Query examples · Sparse Editing · Permanent ID · FAQ · more · Web site
Servers status · Versions · Development · Technical design · Installation · XAPI compatibility layer · Public transport sketch lines · Applications · Source code and issues

Overpass turbo · Wizard · Overpass turbo shortcuts · MapCSS stylesheets · Export to GeoJSON · more · Development · Source code and issues · Web site

Overpass Ultra · Examples · Overpass Ultra extensions · MapLibre stylesheets · URL Params · more · Source code and issues · Web site

Overpass API (mmd fork)
Author:	mmd-osm/Overpass-API/graphs/contributors GitHub
License:	AGPL-3.0 (free of charge)
Platform:	Linux
Version:	0.7.59.120 (unreleased)
Language:	English
Source code:	mmd-osm/Overpass-API GitHub
Programming language:	C++17
A database engine to query the OpenStreetMap data (perf. optimized fork)

Features

See https://github.com/mmd-osm/Overpass-API#readme

Demos

• Achavi Changeset 28764270
• Achavi Changeset 62679580
• Achavi Changeset 45229331
• Around:0 intersection tests

Overpass API 0.7.59_mmd examples

0.7.59_mmd is a heavily patched Overpass prototype version based on release 0.7.56.

Sources: https://github.com/mmd-osm/Overpass-API

New output formats

Export highways in current bounding box in PBF format

[out:pbf];
way[highway]({{bbox}});
(._;>;);
out meta;

Export highways in current bounding box in OPL format

[out:opl];
way[highway]({{bbox}});
(._;>;);
out meta;

With node locations on ways:

[out:opl(geom)];
way[highway]({{bbox}});
out meta geom;

Export pbf

• Highlights AddLocationsToWays pbf extension
• Demo faster attic implementations collect_attic_k and collect_attic_kv

[out:pbf(geom)][bbox:{{bbox}}];

retro("2020-12-20T00:00:00Z") {
  way[highway=residential][lit];
  out geom meta;
}

All supermarkets in Germany in PBF format

[out:pbf];
area[name="Deutschland"];
(nwr[shop=supermarket](area);>;);
out qt;

All xmas features in PBF format (global)

[out:pbf];(nwr["xmas:feature"];>;);out qt;

Regular expressions

PCRE regular expression support

Also see https://www.openstreetmap.org/user/mmd/diary/40197

Nodes with Chinese (Han) characters in current bounding box

// https://www.pcre.org/current/doc/html/pcre2pattern.html

node({{bbox}})["name"~"\\p{Han}"];

out meta;

Identify Tracking parameters

Global query to identify all tag values likely containing tracking parameters. Inspired by Mechanical_Edits/Mateusz_Konieczny_-_bot_account/remove_tracking_parameters2/ by User:Mateusz Konieczny

[timeout:3600];
nwr[~"."~"(fbclid|gclid|campaign_ref|mc_id|utm_source|utm_medium|utm_term|utm_content|utm_campaign|utm_id|gclsrc|dclid|wt.tsrc|WT.tsrc|zanpid|yclid|utm_campain|trkCampaign|mkt_tok|sc_campaign|sc_channel|sc_content|sc_medium|sc_outcome|sc_geo|sc_country|mbid|cmpid|campaign_id|Campaign|fb_action_ids|fb_action_types|fb_ref|fb_source|gs_l|_hsenc|igshid|CampIDMin|CampIDMaj|campaign|Campaign|campaignid|campaignId|adid|adgroupid|refr|referrer|cm_mmc|lw_cmp|CLID|ReferralSource|SourceID|trkid|adjust_creative|partner_slug|y_source|oppartnerid|padid|otppartnerid|ref_device_id|utm_kxconfid|SEO_id|originalReferrer|spMailingID|hsCtaTracking)="];
out geom;

Runtime: 35s vs. 40min on 0.7.57

Emojis in tag values (global)

[timeout:300];

nwr[~"."~"[\\x{0001F600}-\\x{0001F64F}]"];
out tags;

Negative look ahead

Summary: We're looking for address nodes which have some additional information besides the address.

This example returns all nodes in the current bounding box, which have:

• at least one tag containing "addr:"
• at least one tag, which does not contain "addr:"

[bbox:{{bbox}}];
node[~"addr:"~"."][~"^(?!addr:)"~"."];
out;

Foreach statement improvements

Fast foreach

• Foreach w/ union - 1.5mio ways/s

[timeout:3600];
way[building]({{bbox}});
out count;

foreach {

  (._; .result;)->.result;
}

.result out count;

Gas stations per NIST code

(takes more than 2h on 0.7.56, see issue https://github.com/drolbr/Overpass-API/issues/568)

[out:csv(fips, name, total)];

// All counties with FIPS 6-4 county codes
area["nist:fips_code"~"^[0-9]{5}$"];

foreach->.county(
  // Collect all matching features in the current county
  way["amenity"="fuel"](area.county);
  
  // Sum nodes, ways, and relations together and group by county
  make count fips = county.set(t["nist:fips_code"]),
             name = county.set(t["name"]),
             total = count(nwr);
  out;
)

Validating building angles

way({{bbox}})[building](if:!all_vertex(abs(abs(angle()) - 89) < 3));
out geom;

way({{bbox}})[building]
  (if:!all_vertex(abs(sin(angle()/180*6.28318530717959)) < 0.15));

out geom;

Restaurants with nearby public phone (global)

• Restaurant name starts with a "P"
• Phone within 100m distance

[regexp:PCREJIT][out:json][timeout:120];

(
  node["amenity"="restaurant"]["name"~"^P"];
  node["amenity"="bar"]["name"~"^P"];
  node["amenity"="pub"]["name"~"^P"];
  node["amenity"="cafe"]["name"~"^P"];
)->.rest;

nwr["amenity"="telephone"]->.phone;
  
node.rest(around.phone:100);
 
out body;

• 9ba4149: 16s (vs. 2.5 hours without patch)

Starbucks

Story: https://mastodon.social/@amapanda@en.osm.town/113024285038855774

[timeout:3600];
nw["brand:wikidata"="Q37158"];
foreach -> .a {
  nw(around.a:200)["brand:wikidata"="Q37158"];
  if (count(nw) > 4) {
    out center;
  }
}

XAPI style value lists

[out:json][timeout:120];

node[amenity = restaurant | bar | pub | cafe]["name"~"^P"]->.rest;
nwr[amenity=telephone]->.phone;
node.rest(around.phone:100);
 
out body;

• 662c215a: 11s (vs. 2.5 hours without patch)

[out:json]
;
(
  nw[shop=department_store|general|kiosk|mall|supermarket|wholesale|fabric|clothes|baby_goods|
              bag|boutique|fashion|jewelry|leather|sewing|shoes|tailor|watches|second_hand|variety_store|
              charity|hearing_aids|herbalist|beauty|cosmetics|hairdresser|hairdresser_supply|nutrition_supplements|massage|
              optician|perfumery|tattoo|agrarian|bathroom_furnishing|doityourself|electrical|energy|fireplace|
              florist|garden_centre|garden_furniture|gas|glaziery|hardware|houseware|locksmith|paint]
  ({{bbox}});
);
out center;

Queries with very large tag cardinality

Fails on 0.7.62.1 with Query run out of memory using about 2048 MB of RAM., needs at least 12G memory

way[building]                    // 606 282 000 objects
   [source]                      // 220 935 736 objects
   [highway]                     // 232 263 910 objects
   [name]                        //  70 840 162 objects
   ["addr:housenumber"]          //  75 698 229 objects
   ["addr:street"]               //  74 376 041 objects
   ["addr:city"]                 //  57 371 571 objects
   ["addr:postcode"];            //  50 198 913 objects
out meta;

Runtime 24s, max. memory consumption 505M

Negated building and bounding box

[out:json][timeout:100][bbox:50,7,51,8];
(
  way
  ["leisure"="sports_centre"]
  ["sport"="tennis"]
  [!"building"];
);
out center;

Takes 1.3s on 0.7.59.100 (mmd), fails on 0.7.57 with Query run out of memory using about 2048 MB of RAM., with increased maxsize of 3GB the query takes 57s

Large bbox k/v attic queries

Fails on 0.7.56 with Query run out of memory using about 2048 MB of RAM.

[bbox:{{bbox}}];

retro("2020-12-20T00:00:00Z") {
  way[highway=residential][lit=yes];
  out geom meta;
}

Still last editor?

nwr(uid_touched:1);
out count;
nwr._(uid:1);
out count;

Ad hoc area creation

Dolomites on the fly

/*
This shows all mountains (peaks) in the Dolomites.
You may want to use the "zoom onto data" button. =>
*/
[out:json];
// search the relation of the Dolomites
rel
  [place=region]
  ["region:type"="mountain_area"]
  ["name:en"="Dolomites"];
// show the outline
out geom;

// turn the relation into an area
foreach ->.pivot {
  ( 
    way(r.pivot);
    node(w);
  );
  ( make_area [.pivot]; .result;)->.result;
}


node
  [natural=peak]
  (area.result);
out body qt;

Buildings w/o housenumber

Find buildings without housenumber - both on way, as well as inside the way as node.

way[building][!"addr:housenumber"]({{bbox}});

foreach ->.pivot {
  ( 
    .pivot;
    node(w.pivot);
  );

  ( make_area [.pivot]; .result;)->.result;
}

foreach .result -> .area {

  node["addr:housenumber"](area.area);

  if(count(nodes) == 0) {
    way(pivot.area);
    out geom meta;
  }
}

Simplified version, taking advantage of Way based areas:

way[building][!"addr:housenumber"]({{bbox}});

foreach -> .area {

  node["addr:housenumber"](area.area);

  if(count(nodes) == 0) {
    way(pivot.area);
    out geom meta;
  }
}

landuse=residential without buildings

Find landuse=residential ways and relations without any buildings

way[landuse=residential]({{bbox}});

foreach ->.pivot {
  ( 
    .pivot;
    node(w.pivot);
  );

  ( make_area [.pivot]; .result;)->.result;
}

rel[type=multipolygon][landuse=residential]({{bbox}});
foreach ->.pivot {
  ( 
    way(r.pivot);
    node(w);
  );
  ( make_area [.pivot]; .result;)->.result;
}

foreach .result -> .area {

  way[building](area.area);

  if(count(ways) == 0) {
    way(pivot.area);
    out geom meta;
    rel(pivot.area);
    out geom meta;
  }
}

Unnamed landuse residential relations with at least one amenity=bench

rel[type=multipolygon][landuse=residential][!name]({{bbox}});

foreach ->.pivot {
  ( 
    way(r.pivot);
    node(w);
  );
  ( make_area [.pivot]; .result;)->.result;
}


node[amenity=bench](area.result)->.n;

foreach .result -> .r {

  node.n(area.r);
  
  if (count(nodes) > 0) {
    out geom;
    rel(pivot.r);
    out geom;
  }
}

Abusing barrier=line on leisure=pitch ways and relations

way[leisure=pitch]({{bbox}});

foreach ->.pivot {
  ( 
    .pivot;
    node(w.pivot);
  );

  ( make_area [.pivot]; .result;)->.result;
}

rel[type=multipolygon][leisure=pitch]({{bbox}});
foreach ->.pivot {
  ( 
    way(r.pivot);
    node(w);
  );
  ( make_area [.pivot]; .result;)->.result;
}

way[barrier=line](area.result);
out geom;

Simplified version, showing a combination of closed way and relation based areas:

// Combining "closed ways as area" and ...
way[leisure=pitch]({{bbox}})->.result;

// ad-hoc area generation for relations
rel[type=multipolygon][leisure=pitch]({{bbox}});
foreach ->.pivot {
  ( 
    way(r.pivot);
    node(w);
  );
  ( make_area [.pivot]; .result;)->.result;
}

// query for both types in one go
way[barrier=line](area.result);
out geom;

Misc examples

nwr[ele~"^-?[0-9]+(\\.[0-9]+)? ?m$"];
out tags;

node["addr:housenumber"]["addr:street"][!name][!source][!amenity][!attribution][ref];
out  meta;

[timeout:300]
[out:csv(num,key,len)];
area[name="Saarland"];
way[highway=residential][name](area);
for (t["name"])
(
  make stat num=count(ways),key=_.val,len=sum(length());
  out; 
);

Road trip

Featured in blog post https://www.openstreetmap.org/user/mmd/diary/42055

[maxsize:1000000000][timeout:3600][out:pbf];
(
way[highway][highway~"^(trunk|motorway|primary|secondary|tertiary)(_link)?$"]
  (around:20000,50.16634404911624, 8.633879241943358,
   49.01715821614669, 8.374327239990235,
   48.288676648581344, 7.775572357177735,
   47.768872096323875, 7.503656616210939,
   47.53482074712603, 7.630003509521484
 );
node[place=city]
  (around:20000,50.16634404911624, 8.633879241943358,
   49.01715821614669, 8.374327239990235,
   48.288676648581344, 7.775572357177735,
   47.768872096323875, 7.503656616210939,
   47.53482074712603, 7.630003509521484
 );
);

(._;>;);

out meta;

Global electric vehicle charging stations

Global extract of all electric vehicle charging stations as CSV, as mentioned here: https://www.reddit.com/r/openstreetmap/comments/134eveo/turns_out_there_are_90k_electric_vehicle_charging/

[out:csv(type, id, timestamp, ::lat, ::lon, capacity)];
nwr[amenity=charging_station];
convert object id = id(),
               type = type(),
               ::geom = center(geom()),
               timestamp = timestamp(),
               capacity = is_number(t["capacity"]) ? number(t["capacity"]) : 1;
out;

No-U-Turn Relations with from = to member

[timeout:30];

// No-u-turn restrictions mit from = to

rel[restriction=no_u_turn][type=restriction]({{bbox}})
   (if:count_by_role("via") == 1  &&
       count_by_role("from") == 1 &&
       count_by_role("to") == 1);

foreach -> .rel(
  
  way(r.rel:"from") -> .w_from;
  way(r.rel:"to")   -> .w_to;
 
  if ( w_from.min(id()) == w_to.min(id()) )
  (
     .rel out geom meta; 
  );
);

osm2pgsql relation issue

See https://lists.openstreetmap.org/pipermail/dev/2021-February/031047.html, and https://github.com/openstreetmap/osm2pgsql/issues/1394

https://dev.overpass-api.de/blog/total_0_7_56.html#memberpos explains first/last node of a way in more detail.

[timeout:3600];
rel[type=boundary](if:count_by_role("inner") == 0 && count_by_role("outer") == 2) -> .rels;
foreach .rels -> .rel {
  way(r.rel)->.ways;
  node(w.ways:1) ->.firstnodes;
  node(w.ways:-1)->.lastnodes;
  if (ways.count(ways) == 2 && 
      firstnodes.count(nodes) == 1 &&
      lastnodes.count(nodes) == 1) {
    .rel out;
  }
}

Ghost line start- and endpoints

The following query visualizes ghost line start- and endpoints. In most cases, 2 out of 3 nodes are involved, sometimes even 3 out of 3 nodes.

[timeout:3600];
rel({{bbox}})[type=boundary](if:count_by_role("inner") == 0 && count_by_role("outer") == 2) -> .rels;
foreach .rels -> .rel {
  way(r.rel)->.ways;
  node(w.ways:1) ->.firstnodes;
  node(w.ways:-1)->.lastnodes;
  if (ways.count(ways) == 2 && 
      firstnodes.count(nodes) == 1 &&
      lastnodes.count(nodes) == 1) {
    node(w.ways:-1);out;
    node(w.ways:2);out;
    .rel out geom;
  }
}

Suspects - Overpass edition

See OSMsuspects! for details

All queries in this section should also run on overpass-api.de, unless stated otherwise (performance might not be ideal).

addr:housenumber check

[timeout:300][bbox:{{bbox}}];

// https://wiki.openstreetmap.org/wiki/DE:OSMsuspects!#Pr%C3%BCfung_addr:housenumber

nw["addr:housenumber"]
    ["addr:housenumber"!~"(^[1-9][0-9]{0,3}\\s{0,1}[a-zA-Z]{0,1})$"]
    ["addr:housenumber"!~"(^[1-9][0-9]{0,3}[-/][1-9][0-9]{0,3})$"]
    ["addr:housenumber"!~"(^[1-9][0-9]{0,3}\\s{1,1}[-/]\s{1,1}[1-9][0-9]{0,3})$"]
    ["addr:housenumber"!~"(^[1-9][0-9]{0,3}\\s{1,1}[1-9][0-9]{0,3}[/][1-9][0-9]{0,3})$"]
    ["addr:housenumber"!~"(^[A-Z]{1,2}\\s{0,1}[1-9][0-9]{0,3})$"];
out geom;

addr:street check

Check if addr:street node/way has closely (<500m) highway with the same name in the current bounding box. If highway doesn't exist, also check for nearby place nodes.

Very similar to OSMI: https://tools.geofabrik.de/osmi/?view=addresses&lon=13.05266&lat=48.77358&zoom=16&opacity=0.56&overlays=no_addr_street,street_not_found,place_not_found,misformatted_housenumber_lenient,nodes_with_addresses_interpolated

[timeout:300];

// Prefetch places, ways and addr:streets in bounding box
node[place][name]({{bbox}})->.places;
way[highway][name]({{bbox}})->.ways;
nw["addr:street"]({{bbox}})->.addrstreets;

if (count(ways) <= 10000) {    // limit query size: up to 10'000 addr:street only

// iterate over all addr:street ways/nodes  
  
foreach .addrstreets -> .addrstreet {

// nearby highway with same name?
  way.ways(around.addrstreet:500)(if:t["name"] == addrstreet.u(t["addr:street"]));

  if (count(ways) == 0) {
    
    // no -> maybe nearby place node with same name?
    node.places(around.addrstreet:500)(if:t["name"] == addrstreet.u(t["addr:street"]));
    
    // no -> print out addr:street
    if(count(nodes) == 0) {
      .addrstreet out geom; 
    }
  }
}

}

{{style:
   
   node, way { 
     text: eval("tag('addr:street') . ' ' . tag('addr:housenumber')");
   }   
   
}}

Duplicates

Same addr:housenumber + addr:street + addr_city within 1000m (simplified)

[timeout:300];

  
nw["addr:housenumber"]["addr:street"]({{bbox}});


for -> .obj (t["addr:housenumber"] + "|" + t["addr:street"] + "|" + t["addr:city"])
(
  if (obj.count(nw) > 1) {
    
       foreach .obj -> .x {
   
         nw.obj(around.x:1000)
           (if:t["addr:housenumber"] == x.u(t["addr:housenumber"]) &&
               t["addr:street"] == x.u(t["addr:street"]) &&
               t["addr:city"] == x.u(t["addr:city"]));

         if (count(nw) > 1) {
          (._; .result;)->.result;
         }
 
       }
    
  }
);
  
.result out geom;

   
{{style:
   
   node, way { 
     text: eval("tag('addr:street') . ' ' . tag('addr:housenumber')");
   }   
   
}}

postal code test

[timeout:300];
rel["postal_code"][boundary=postal_code]({{bbox}});

map_to_area -> .result;

foreach .result -> .r {

nw
  ["addr:postcode"]
  (if:t["addr:postcode"] != r.set(t["postal_code"]])
  (area.r);

   out center;
   out geom;

   if (count(nw) > 0) {
     rel(pivot.r);
     out geom;
   }
  
};
   
{{style:
   
   relation[boundary=postal_code] { 
     opacity:0.5;
     color: blue;
     fill-color:grey;
     text: eval("tag('postal_code')");
   }   
   
}}

addr:housenumber mismatch to enclosing building outline

// addr:housenumber mismatch to enclosing building outline

node({{bbox}})["addr:housenumber"];
is_in;
way._["addr:housenumber"]->.ways;

foreach .ways -> .way {
   node(area.way)["addr:housenumber"]
     (if:way.u(t["addr:housenumber"]) != t["addr:housenumber"]);
   out;
  
}

addr:housenumber + addr:street without city or postcode

// addr:housenumber + addr:street without city or postcode

nw({{bbox}})["addr:housenumber"]["addr:street"]
   (if:is_tag("addr:city") + is_tag("addr:postcode") == 0);
out geom;

postal code test( Overpass API 0.7.59_mmd only)

[timeout:300];
rel["postal_code"][boundary=postal_code]({{bbox}});

foreach ->.pivot {
  ( 
    way(r.pivot);
    node(w);
  );
  ( make_area [.pivot]; .result;)->.result;
}

foreach .result -> .r {

nw
  ["addr:postcode"]
  (if:t["addr:postcode"] != r.set(t["postal_code"]])
  (area.r);

   out center;
   out geom;

   if (count(nw) > 0) {
     rel(pivot.r);
     out geom;
   }
  
};
   
{{style:
   
   relation[boundary=postal_code] { 
     opacity:0.5;
     color: blue;
     fill-color:grey;
     text: eval("tag('postal_code')");
   }   
   
}}

Installation

Based on https://overpass-api.de/full_installation.html, only deltas described here.

Compiling

Additional dependencies on FastCGI, BZ2 and ICU.

git clone https://github.com/mmd-osm/Overpass-API.git
cd Overpass-API
git checkout test759
git submodule update --init

sudo apt-get update -qq || true
sudo apt-get install -y expat libexpat1-dev liblz4-dev libfcgi-dev zlib1g-dev libbz2-dev libicu-dev libfmt-dev libpcre2-dev
  
pushd src/
chmod u+x test-bin/*.sh
autoscan
aclocal
autoheader
libtoolize
automake --add-missing
autoconf
popd
mkdir -p build
cd build
../src/configure CXXFLAGS="-Werror=implicit-function-declaration  -D_FORTIFY_SOURCE=2 -fexceptions -fpie -Wl,-pie -fpic -shared -fstack-protector-strong -Wl,--no-as-needed -pipe -Wl,-z,defs -Wl,-z,now -Wl,-z,relro -fno-omit-frame-pointer -flto -fwhole-program -lfmt -O2" LDFLAGS="-flto -fwhole-program -lpcre2-8" --prefix=$EXEC_DIR --enable-lz4 --enable-fastcgi --enable-tests
make V=0 -j3
make install

configure CXXFLAGS="-I/usr/local/include -D_FORTIFY_SOURCE=2 -fexceptions -ggdb -fpie -Wl,-pie -fpic -shared -fstack-protector-strong -Wl,--no-as-needed -pipe -Wl,-z,defs -Wl,-z,now -Wl,-z,relro -fwhole-program -flto=1 -fno-omit-frame-pointer -std=c++17 -O2  -lfmt -march=skylake"  --enable-lz4 --enable-fastcgi LDFLAGS="-fwhole-program -flto=1"

(optionally use tcmalloc)

FastCGI

supervisord config

/etc/supervisor/conf.d/overpass.conf

[fcgi-program:interpreter]
socket=unix:///var/run/interpreter.socket
socket_owner=www-data
socket_mode=0660
environment=
    OVERPASS_FCGI_MAX_REQUESTS=10000,
    OVERPASS_FCGI_MAX_ELAPSED_TIME=900,
    OVERPASS_REGEXP_ENGINE="PCREJIT"
command=/home/user/osm3s/fcgi-bin/interpreter
numprocs=6
priority=999
process_name=%(program_name)s_%(process_num)02d
user=www-data
autorestart=true
autostart=true
startsecs=1
startretries=3
stopsignal=QUIT
stopwaitsecs=10
redirect_stderr=true
stdout_logfile=/var/log/interpreter.log
stdout_logfile_maxbytes=10MB

Apache config

Depends on mod_proxy_fcgi

Forward calls to /api/interpreter to local socket managed by supervisord.

         ProxyPass /api/interpreter unix:///var/run/interpreter.socket|fcgi://localhost/api/interpreter