User:mmd/Overpass API - experimental corner
Servers status · Versions · Development · Technical design · Installation · XAPI compatibility layer · Public transport sketch lines · Applications · Source code and issuesOverpass turbo · Wizard · Overpass turbo shortcuts · MapCSS stylesheets · Export to GeoJSON · more · Development · Source code and issues · Web siteOverpass Ultra · Examples · Overpass Ultra extensions · MapLibre stylesheets · URL Params · more · Source code and issues · Web site
Overpass API (mmd fork) | ||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Author: | mmd-osm/Overpass-API/graphs/contributors GitHub | |||||||||||||||||||||||||||||||||||||||||||
License: | GNU AGPL v3 (free of charge) | |||||||||||||||||||||||||||||||||||||||||||
Platform: | Linux | |||||||||||||||||||||||||||||||||||||||||||
Version: | 0.7.59.120 (unreleased) | |||||||||||||||||||||||||||||||||||||||||||
Language: | English | |||||||||||||||||||||||||||||||||||||||||||
Source code: | mmd-osm/Overpass-API GitHub | |||||||||||||||||||||||||||||||||||||||||||
Programming language: | C++17 | |||||||||||||||||||||||||||||||||||||||||||
A database engine to query the OpenStreetMap data (perf. optimized fork) |
||||||||||||||||||||||||||||||||||||||||||||
|
Features
See https://github.com/mmd-osm/Overpass-API#readme
Demos
- Achavi Changeset 28764270
- Achavi Changeset 62679580
- Achavi Changeset 45229331
- Around:0 intersection tests
Overpass API 0.7.59_mmd examples
0.7.59_mmd is a heavily patched Overpass prototype version based on release 0.7.56.
Sources: https://github.com/mmd-osm/Overpass-API
Examples on this page are not supported on the official Overpass API instance at overpass-api.de! unless indicated otherwise |
All dev instance demos require an API key moving forward |
New output formats
Export highways in current bounding box in PBF format
[out:pbf];
way[highway]({{bbox}});
(._;>;);
out meta;
Export highways in current bounding box in OPL format
[out:opl];
way[highway]({{bbox}});
(._;>;);
out meta;
With node locations on ways:
[out:opl(geom)];
way[highway]({{bbox}});
out meta geom;
Export pbf
- Highlights AddLocationsToWays pbf extension
- Demo faster attic implementations collect_attic_k and collect_attic_kv
[out:pbf(geom)][bbox:{{bbox}}];
retro("2020-12-20T00:00:00Z") {
way[highway=residential][lit];
out geom meta;
}
All supermarkets in Germany in PBF format
[out:pbf];
area[name="Deutschland"];
(nwr[shop=supermarket](area);>;);
out qt;
All xmas features in PBF format (global)
[out:pbf];(nwr["xmas:feature"];>;);out qt;
Regular expressions
PCRE regular expression support
Also see https://www.openstreetmap.org/user/mmd/diary/40197
Nodes with Chinese (Han) characters in current bounding box
// https://www.pcre.org/current/doc/html/pcre2pattern.html
node({{bbox}})["name"~"\\p{Han}"];
out meta;
Identify Tracking parameters
Global query to identify all tag values likely containing tracking parameters. Inspired by Mechanical_Edits/Mateusz_Konieczny_-_bot_account/remove_tracking_parameters2/ by User:Mateusz Konieczny
[timeout:3600];
nwr[~"."~"(fbclid|gclid|campaign_ref|mc_id|utm_source|utm_medium|utm_term|utm_content|utm_campaign|utm_id|gclsrc|dclid|wt.tsrc|WT.tsrc|zanpid|yclid|utm_campain|trkCampaign|mkt_tok|sc_campaign|sc_channel|sc_content|sc_medium|sc_outcome|sc_geo|sc_country|mbid|cmpid|campaign_id|Campaign|fb_action_ids|fb_action_types|fb_ref|fb_source|gs_l|_hsenc|igshid|CampIDMin|CampIDMaj|campaign|Campaign|campaignid|campaignId|adid|adgroupid|refr|referrer|cm_mmc|lw_cmp|CLID|ReferralSource|SourceID|trkid|adjust_creative|partner_slug|y_source|oppartnerid|padid|otppartnerid|ref_device_id|utm_kxconfid|SEO_id|originalReferrer|spMailingID|hsCtaTracking)="];
out geom;
Runtime: 35s vs. 40min on 0.7.57
Emojis in tag values (global)
[timeout:300];
nwr[~"."~"[\\x{0001F600}-\\x{0001F64F}]"];
out tags;
Negative look ahead
Summary: We're looking for address nodes which have some additional information besides the address.
This example returns all nodes in the current bounding box, which have:
- at least one tag containing "addr:"
- at least one tag, which does not contain "addr:"
[bbox:{{bbox}}];
node[~"addr:"~"."][~"^(?!addr:)"~"."];
out;
Foreach statement improvements
Fast foreach
- Foreach w/ union - 1.5mio ways/s
[timeout:3600];
way[building]({{bbox}});
out count;
foreach {
(._; .result;)->.result;
}
.result out count;
Gas stations per NIST code
(takes more than 2h on 0.7.56, see issue https://github.com/drolbr/Overpass-API/issues/568)
[out:csv(fips, name, total)];
// All counties with FIPS 6-4 county codes
area["nist:fips_code"~"^[0-9]{5}$"];
foreach->.county(
// Collect all matching features in the current county
way["amenity"="fuel"](area.county);
// Sum nodes, ways, and relations together and group by county
make count fips = county.set(t["nist:fips_code"]),
name = county.set(t["name"]),
total = count(nwr);
out;
)
Validating building angles
way({{bbox}})[building](if:!all_vertex(abs(abs(angle()) - 89) < 3));
out geom;
way({{bbox}})[building]
(if:!all_vertex(abs(sin(angle()/180*6.28318530717959)) < 0.15));
out geom;
Restaurants with nearby public phone (global)
- Restaurant name starts with a "P"
- Phone within 100m distance
[regexp:PCREJIT][out:json][timeout:120];
(
node["amenity"="restaurant"]["name"~"^P"];
node["amenity"="bar"]["name"~"^P"];
node["amenity"="pub"]["name"~"^P"];
node["amenity"="cafe"]["name"~"^P"];
)->.rest;
nwr["amenity"="telephone"]->.phone;
node.rest(around.phone:100);
out body;
- 9ba4149: 16s (vs. 2.5 hours without patch)
Starbucks
Story: https://mastodon.social/@amapanda@en.osm.town/113024285038855774
[timeout:3600];
nw["brand:wikidata"="Q37158"];
foreach -> .a {
nw(around.a:200)["brand:wikidata"="Q37158"];
if (count(nw) > 4) {
out center;
}
}
XAPI style value lists
[out:json][timeout:120];
node[amenity = restaurant | bar | pub | cafe]["name"~"^P"]->.rest;
nwr[amenity=telephone]->.phone;
node.rest(around.phone:100);
out body;
- 662c215a: 11s (vs. 2.5 hours without patch)
[out:json]
;
(
nw[shop=department_store|general|kiosk|mall|supermarket|wholesale|fabric|clothes|baby_goods|
bag|boutique|fashion|jewelry|leather|sewing|shoes|tailor|watches|second_hand|variety_store|
charity|hearing_aids|herbalist|beauty|cosmetics|hairdresser|hairdresser_supply|nutrition_supplements|massage|
optician|perfumery|tattoo|agrarian|bathroom_furnishing|doityourself|electrical|energy|fireplace|
florist|garden_centre|garden_furniture|gas|glaziery|hardware|houseware|locksmith|paint]
({{bbox}});
);
out center;
Queries with very large tag cardinality
Fails on 0.7.62.1 with Query run out of memory using about 2048 MB of RAM., needs at least 12G memory
way[building] // 606 282 000 objects
[source] // 220 935 736 objects
[highway] // 232 263 910 objects
[name] // 70 840 162 objects
["addr:housenumber"] // 75 698 229 objects
["addr:street"] // 74 376 041 objects
["addr:city"] // 57 371 571 objects
["addr:postcode"]; // 50 198 913 objects
out meta;
Runtime 24s, max. memory consumption 505M
Negated building and bounding box
[out:json][timeout:100][bbox:50,7,51,8];
(
way
["leisure"="sports_centre"]
["sport"="tennis"]
[!"building"];
);
out center;
Takes 1.3s on 0.7.59.100 (mmd), fails on 0.7.57 with Query run out of memory using about 2048 MB of RAM., with increased maxsize of 3GB the query takes 57s
Large bbox k/v attic queries
Fails on 0.7.56 with Query run out of memory using about 2048 MB of RAM.
[bbox:{{bbox}}];
retro("2020-12-20T00:00:00Z") {
way[highway=residential][lit=yes];
out geom meta;
}
Still last editor?
nwr(uid_touched:1);
out count;
nwr._(uid:1);
out count;
Ad hoc area creation
Dolomites on the fly
/*
This shows all mountains (peaks) in the Dolomites.
You may want to use the "zoom onto data" button. =>
*/
[out:json];
// search the relation of the Dolomites
rel
[place=region]
["region:type"="mountain_area"]
["name:en"="Dolomites"];
// show the outline
out geom;
// turn the relation into an area
foreach ->.pivot {
(
way(r.pivot);
node(w);
);
( make_area [.pivot]; .result;)->.result;
}
node
[natural=peak]
(area.result);
out body qt;
Buildings w/o housenumber
Find buildings without housenumber - both on way, as well as inside the way as node.
way[building][!"addr:housenumber"]({{bbox}});
foreach ->.pivot {
(
.pivot;
node(w.pivot);
);
( make_area [.pivot]; .result;)->.result;
}
foreach .result -> .area {
node["addr:housenumber"](area.area);
if(count(nodes) == 0) {
way(pivot.area);
out geom meta;
}
}
Simplified version, taking advantage of Way based areas:
way[building][!"addr:housenumber"]({{bbox}});
foreach -> .area {
node["addr:housenumber"](area.area);
if(count(nodes) == 0) {
way(pivot.area);
out geom meta;
}
}
landuse=residential without buildings
Find landuse=residential ways and relations without any buildings
way[landuse=residential]({{bbox}});
foreach ->.pivot {
(
.pivot;
node(w.pivot);
);
( make_area [.pivot]; .result;)->.result;
}
rel[type=multipolygon][landuse=residential]({{bbox}});
foreach ->.pivot {
(
way(r.pivot);
node(w);
);
( make_area [.pivot]; .result;)->.result;
}
foreach .result -> .area {
way[building](area.area);
if(count(ways) == 0) {
way(pivot.area);
out geom meta;
rel(pivot.area);
out geom meta;
}
}
Unnamed landuse residential relations with at least one amenity=bench
rel[type=multipolygon][landuse=residential][!name]({{bbox}});
foreach ->.pivot {
(
way(r.pivot);
node(w);
);
( make_area [.pivot]; .result;)->.result;
}
node[amenity=bench](area.result)->.n;
foreach .result -> .r {
node.n(area.r);
if (count(nodes) > 0) {
out geom;
rel(pivot.r);
out geom;
}
}
Abusing barrier=line on leisure=pitch ways and relations
way[leisure=pitch]({{bbox}});
foreach ->.pivot {
(
.pivot;
node(w.pivot);
);
( make_area [.pivot]; .result;)->.result;
}
rel[type=multipolygon][leisure=pitch]({{bbox}});
foreach ->.pivot {
(
way(r.pivot);
node(w);
);
( make_area [.pivot]; .result;)->.result;
}
way[barrier=line](area.result);
out geom;
Simplified version, showing a combination of closed way and relation based areas:
// Combining "closed ways as area" and ...
way[leisure=pitch]({{bbox}})->.result;
// ad-hoc area generation for relations
rel[type=multipolygon][leisure=pitch]({{bbox}});
foreach ->.pivot {
(
way(r.pivot);
node(w);
);
( make_area [.pivot]; .result;)->.result;
}
// query for both types in one go
way[barrier=line](area.result);
out geom;
Misc examples
nwr[ele~"^-?[0-9]+(\\.[0-9]+)? ?m$"];
out tags;
node["addr:housenumber"]["addr:street"][!name][!source][!amenity][!attribution][ref];
out meta;
[timeout:300]
[out:csv(num,key,len)];
area[name="Saarland"];
way[highway=residential][name](area);
for (t["name"])
(
make stat num=count(ways),key=_.val,len=sum(length());
out;
);
Road trip
Featured in blog post https://www.openstreetmap.org/user/mmd/diary/42055
[maxsize:1000000000][timeout:3600][out:pbf];
(
way[highway][highway~"^(trunk|motorway|primary|secondary|tertiary)(_link)?$"]
(around:20000,50.16634404911624, 8.633879241943358,
49.01715821614669, 8.374327239990235,
48.288676648581344, 7.775572357177735,
47.768872096323875, 7.503656616210939,
47.53482074712603, 7.630003509521484
);
node[place=city]
(around:20000,50.16634404911624, 8.633879241943358,
49.01715821614669, 8.374327239990235,
48.288676648581344, 7.775572357177735,
47.768872096323875, 7.503656616210939,
47.53482074712603, 7.630003509521484
);
);
(._;>;);
out meta;
Global electric vehicle charging stations
Global extract of all electric vehicle charging stations as CSV, as mentioned here: https://www.reddit.com/r/openstreetmap/comments/134eveo/turns_out_there_are_90k_electric_vehicle_charging/
[out:csv(type, id, timestamp, ::lat, ::lon, capacity)];
nwr[amenity=charging_station];
convert object id = id(),
type = type(),
::geom = center(geom()),
timestamp = timestamp(),
capacity = is_number(t["capacity"]) ? number(t["capacity"]) : 1;
out;
No-U-Turn Relations with from = to member
[timeout:30];
// No-u-turn restrictions mit from = to
rel[restriction=no_u_turn][type=restriction]({{bbox}})
(if:count_by_role("via") == 1 &&
count_by_role("from") == 1 &&
count_by_role("to") == 1);
foreach -> .rel(
way(r.rel:"from") -> .w_from;
way(r.rel:"to") -> .w_to;
if ( w_from.min(id()) == w_to.min(id()) )
(
.rel out geom meta;
);
);
osm2pgsql relation issue
See https://lists.openstreetmap.org/pipermail/dev/2021-February/031047.html, and https://github.com/openstreetmap/osm2pgsql/issues/1394
https://dev.overpass-api.de/blog/total_0_7_56.html#memberpos explains first/last node of a way in more detail.
[timeout:3600];
rel[type=boundary](if:count_by_role("inner") == 0 && count_by_role("outer") == 2) -> .rels;
foreach .rels -> .rel {
way(r.rel)->.ways;
node(w.ways:1) ->.firstnodes;
node(w.ways:-1)->.lastnodes;
if (ways.count(ways) == 2 &&
firstnodes.count(nodes) == 1 &&
lastnodes.count(nodes) == 1) {
.rel out;
}
}
Ghost line start- and endpoints
The following query visualizes ghost line start- and endpoints. In most cases, 2 out of 3 nodes are involved, sometimes even 3 out of 3 nodes.
[timeout:3600];
rel({{bbox}})[type=boundary](if:count_by_role("inner") == 0 && count_by_role("outer") == 2) -> .rels;
foreach .rels -> .rel {
way(r.rel)->.ways;
node(w.ways:1) ->.firstnodes;
node(w.ways:-1)->.lastnodes;
if (ways.count(ways) == 2 &&
firstnodes.count(nodes) == 1 &&
lastnodes.count(nodes) == 1) {
node(w.ways:-1);out;
node(w.ways:2);out;
.rel out geom;
}
}
Suspects - Overpass edition
See OSMsuspects! for details
All queries in this section should also run on overpass-api.de, unless stated otherwise (performance might not be ideal).
addr:housenumber check
[timeout:300][bbox:{{bbox}}];
// https://wiki.openstreetmap.org/wiki/DE:OSMsuspects!#Pr%C3%BCfung_addr:housenumber
nw["addr:housenumber"]
["addr:housenumber"!~"(^[1-9][0-9]{0,3}\\s{0,1}[a-zA-Z]{0,1})$"]
["addr:housenumber"!~"(^[1-9][0-9]{0,3}[-/][1-9][0-9]{0,3})$"]
["addr:housenumber"!~"(^[1-9][0-9]{0,3}\\s{1,1}[-/]\s{1,1}[1-9][0-9]{0,3})$"]
["addr:housenumber"!~"(^[1-9][0-9]{0,3}\\s{1,1}[1-9][0-9]{0,3}[/][1-9][0-9]{0,3})$"]
["addr:housenumber"!~"(^[A-Z]{1,2}\\s{0,1}[1-9][0-9]{0,3})$"];
out geom;
addr:street check
Check if addr:street node/way has closely (<500m) highway with the same name in the current bounding box. If highway doesn't exist, also check for nearby place nodes.
[timeout:300];
// Prefetch places, ways and addr:streets in bounding box
node[place][name]({{bbox}})->.places;
way[highway][name]({{bbox}})->.ways;
nw["addr:street"]({{bbox}})->.addrstreets;
if (count(ways) <= 10000) { // limit query size: up to 10'000 addr:street only
// iterate over all addr:street ways/nodes
foreach .addrstreets -> .addrstreet {
// nearby highway with same name?
way.ways(around.addrstreet:500)(if:t["name"] == addrstreet.u(t["addr:street"]));
if (count(ways) == 0) {
// no -> maybe nearby place node with same name?
node.places(around.addrstreet:500)(if:t["name"] == addrstreet.u(t["addr:street"]));
// no -> print out addr:street
if(count(nodes) == 0) {
.addrstreet out geom;
}
}
}
}
{{style:
node, way {
text: eval("tag('addr:street') . ' ' . tag('addr:housenumber')");
}
}}
Duplicates
Same addr:housenumber + addr:street + addr_city within 1000m (simplified)
[timeout:300];
nw["addr:housenumber"]["addr:street"]({{bbox}});
for -> .obj (t["addr:housenumber"] + "|" + t["addr:street"] + "|" + t["addr:city"])
(
if (obj.count(nw) > 1) {
foreach .obj -> .x {
nw.obj(around.x:1000)
(if:t["addr:housenumber"] == x.u(t["addr:housenumber"]) &&
t["addr:street"] == x.u(t["addr:street"]) &&
t["addr:city"] == x.u(t["addr:city"]));
if (count(nw) > 1) {
(._; .result;)->.result;
}
}
}
);
.result out geom;
{{style:
node, way {
text: eval("tag('addr:street') . ' ' . tag('addr:housenumber')");
}
}}
postal code test
[timeout:300];
rel["postal_code"][boundary=postal_code]({{bbox}});
map_to_area -> .result;
foreach .result -> .r {
nw
["addr:postcode"]
(if:t["addr:postcode"] != r.set(t["postal_code"]])
(area.r);
out center;
out geom;
if (count(nw) > 0) {
rel(pivot.r);
out geom;
}
};
{{style:
relation[boundary=postal_code] {
opacity:0.5;
color: blue;
fill-color:grey;
text: eval("tag('postal_code')");
}
}}
addr:housenumber mismatch to enclosing building outline
// addr:housenumber mismatch to enclosing building outline
node({{bbox}})["addr:housenumber"];
is_in;
way._["addr:housenumber"]->.ways;
foreach .ways -> .way {
node(area.way)["addr:housenumber"]
(if:way.u(t["addr:housenumber"]) != t["addr:housenumber"]);
out;
}
addr:housenumber + addr:street without city or postcode
// addr:housenumber + addr:street without city or postcode
nw({{bbox}})["addr:housenumber"]["addr:street"]
(if:is_tag("addr:city") + is_tag("addr:postcode") == 0);
out geom;
postal code test( Overpass API 0.7.59_mmd only)
[timeout:300];
rel["postal_code"][boundary=postal_code]({{bbox}});
foreach ->.pivot {
(
way(r.pivot);
node(w);
);
( make_area [.pivot]; .result;)->.result;
}
foreach .result -> .r {
nw
["addr:postcode"]
(if:t["addr:postcode"] != r.set(t["postal_code"]])
(area.r);
out center;
out geom;
if (count(nw) > 0) {
rel(pivot.r);
out geom;
}
};
{{style:
relation[boundary=postal_code] {
opacity:0.5;
color: blue;
fill-color:grey;
text: eval("tag('postal_code')");
}
}}
Installation
Based on https://overpass-api.de/full_installation.html, only deltas described here.
Compiling
Not released for general usage! |
Additional dependencies on FastCGI, BZ2 and ICU.
git clone https://github.com/mmd-osm/Overpass-API.git
cd Overpass-API
git checkout test759
git submodule update --init
sudo apt-get update -qq || true
sudo apt-get install -y expat libexpat1-dev liblz4-dev libfcgi-dev zlib1g-dev libbz2-dev libicu-dev libfmt-dev libpcre2-dev
pushd src/
chmod u+x test-bin/*.sh
autoscan
aclocal
autoheader
libtoolize
automake --add-missing
autoconf
popd
mkdir -p build
cd build
../src/configure CXXFLAGS="-Werror=implicit-function-declaration -D_FORTIFY_SOURCE=2 -fexceptions -fpie -Wl,-pie -fpic -shared -fstack-protector-strong -Wl,--no-as-needed -pipe -Wl,-z,defs -Wl,-z,now -Wl,-z,relro -fno-omit-frame-pointer -flto -fwhole-program -lfmt -O2" LDFLAGS="-flto -fwhole-program -lpcre2-8" --prefix=$EXEC_DIR --enable-lz4 --enable-fastcgi --enable-tests
make V=0 -j3
make install
configure CXXFLAGS="-I/usr/local/include -D_FORTIFY_SOURCE=2 -fexceptions -ggdb -fpie -Wl,-pie -fpic -shared -fstack-protector-strong -Wl,--no-as-needed -pipe -Wl,-z,defs -Wl,-z,now -Wl,-z,relro -fwhole-program -flto=1 -fno-omit-frame-pointer -std=c++17 -O2 -lfmt -march=skylake" --enable-lz4 --enable-fastcgi LDFLAGS="-fwhole-program -flto=1"
(optionally use tcmalloc)
FastCGI
supervisord config
/etc/supervisor/conf.d/overpass.conf
[fcgi-program:interpreter]
socket=unix:///var/run/interpreter.socket
socket_owner=www-data
socket_mode=0660
environment=
OVERPASS_FCGI_MAX_REQUESTS=10000,
OVERPASS_FCGI_MAX_ELAPSED_TIME=900,
OVERPASS_REGEXP_ENGINE="PCREJIT"
command=/home/user/osm3s/fcgi-bin/interpreter
numprocs=6
priority=999
process_name=%(program_name)s_%(process_num)02d
user=www-data
autorestart=true
autostart=true
startsecs=1
startretries=3
stopsignal=QUIT
stopwaitsecs=10
redirect_stderr=true
stdout_logfile=/var/log/interpreter.log
stdout_logfile_maxbytes=10MB
Apache config
Depends on mod_proxy_fcgi
Forward calls to /api/interpreter to local socket managed by supervisord.
ProxyPass /api/interpreter unix:///var/run/interpreter.socket|fcgi://localhost/api/interpreter