User:Breki/Overpass API Installation
Introduction
This document tries to give a step-by-step guidance on installing Overpass API on your own computer. The installation procedure described was done on Xubuntu 16.04.2 (64 bit desktop image version) using Overpass API v0.7.54.
The procedure is "opinionated" and follows a specific order of installation steps and is not mean to be a general "how-to" on Overpass API installations. I just wanted to describe in detail how I set up Overpass API (after a long and painful process of learning and troubleshooting) that actually works, without many alternative ways of configuring the system and Overpass API.
Since I am a Linux newbie, I cannot guarantee all the steps in this procedure are the "right way" of doing things on Linux. If you spot any issues, please make a note in the Talk page. And please do not modify the contents of the main page. Thanks!
Installing Prerequisite Packages
These packages are needed so you can compile Overpass API package and run it. Apache web server package is also included.
sudo apt-get update sudo apt-get install g++ make expat libexpat1-dev zlib1g-dev apache2 -y
Preparing Overpass API Binaries
Downloading Overpass API Tarball And Compiling The Source Code
We will put all Overpass files into the /home/igor/overpass (~/overpass) directory. For simplicity reasons, Overpass binaries will be built inside the sources directory.
cd Downloads wget http://dev.overpass-api.de/releases/osm-3s_v0.7.54.tar.gz cd .. mkdir overpass cd overpass mkdir db tar -zxvf ../Downloads/osm-3s_v0.7.54.tar.gz cd osm-3s_v0.7.54 ./configure --prefix="`pwd`" make
Enabling Access To Overpass API Package Files
After the compilation, we make sure the scripts will have the necessary access to the Overpass API package files:
chmod -R 755 ~/overpass
Trying It Out With A Sample OSM Dataset (Optional)
This section describes how you can try Overpass API with a smaller sample OSM dataset (if you do not intend to use the whole Planet or just want to see if the pieces of the puzzle fit together).
Downloading Sample OSM Data
We are using a Isle of Man sample extract from Geofabrik since it is small and (mostly) self-contained.
cd ~/Downloads wget http://download.geofabrik.de/europe/isle-of-man-latest.osm.bz2 cd ..
Populating Overpass API Database With The Small Sample
For testing reasons, we will do the populating in-process since it should not take too long for such a small dataset:
cd overpass osm-3s_v0.7.54/bin/init_osm3s.sh ../Downloads/isle-of-man-latest.osm.bz2 db osm-3s_v0.7.54
Running A Sample Query
Now we are ready to run a sample query on the imported data. Enter command
osm-3s_v0.7.54/bin/osm3s_query --db-dir=db
... and then copy & paste this sample query into the prompt:
node["amenity"="post_box"]; out skel;
... and press Ctrl+D to start the query. It should return skeleton OSM data about all post box nodes on Isle of Man.
Populating Overpass API Database (Whole Planet)
Cleaning Up The db Directory
Clean the existing db directory (if you populated it using the sample dataset in the previous section, otherwise skip to the next subsection)
cd ~/overpass rm -rf db mkdir db
Populating The Database Via Cloning
Now we need to populate the database via cloning from the dev server. Note that this can take a long time (from several hours to several days, depending on your hardware, internet connection and other things).
You can do this directly in-process (which is easier to follow since the output/progress of the command is immediately visible in the terminal):
cd ~/overpass osm-3s_v0.7.54/bin/download_clone.sh --db-dir=db --source=http://dev.overpass-api.de/api_drolbr/ --meta=no
... or out-of-process using nohup command so the process can continue to run even if the user logs out:
cd ~/overpass nohup osm-3s_v0.7.54/bin/download_clone.sh --db-dir=db --source=http://dev.overpass-api.de/api_drolbr/ --meta=no &
(note that the ending & character means the command will run in the background).
The database will be stored in the ~/overpass/db directory.
Configuring The Dispatcher Daemon
Configuring systemd
Before configuring the systemd service, we need to tell systemd not to delete shared memory files created by non-system user accounts that are not logged in (since we will use a non-system user account igor to run system service(s)). To to that, first open /etc/systemd/logind.conf file in the editor:
sudo nano /etc/systemd/logind.conf
and uncomment the line RemoveIPC=yes, change it to RemoveIPC=no, save the file and reboot the system.
Preparing The Dispatcher Daemon Configuration
First we need to prepare a systemd configuration file (called overpass.service) for the dispatcher daemon:
sudo nano /lib/systemd/system/overpass.service
Paste the following into the text editor once it opens and save the file. Note that you need to change the user account the service will be running as (the User directive) and the various parameters that point to the /home/igor/overpass directory:
[Unit] Description=Overpass init agent [Service] Type=simple User=igor ExecStartPre=/bin/rm /home/igor/overpass/db/osm3s* -f --verbose ExecStartPre=/bin/rm /dev/shm/osm3s* -f --verbose ExecStart=/home/igor/overpass/osm-3s_v0.7.54/bin/dispatcher --osm-base --db-dir=/home/igor/overpass/db [Install] WantedBy=multi-user.target
Enabling The Dispatcher Daemon
Now enable and start the Overpass service:
sudo systemctl enable overpass sudo systemctl start overpass
Troubleshooting The Dispatcher Daemon Problems
You can check the status of the service with the following command:
sudo systemctl status overpass | more
It should report the service as loaded and active (running), and it should also report ExecStartPre as successfully executed, something like the following output:
overpass.service - Overpass init agent Loaded: loaded (/lib/systemd/system/overpass.service; enabled; vendor preset: enabled) Active: active (running) since ned 2017-07-23 10:14:27 CEST; 1h 47min ago Process: 4282 ExecStartPre=/bin/rm /dev/shm/osm3s* -f --verbose (code=exited, status=0/SUCCESS) Process: 4280 ExecStartPre=/bin/rm /home/igor/overpass/db/osm3s* -f --verbose (code=exited, status=0/SUCCESS) Main PID: 4285 (dispatcher) CGroup: /system.slice/overpass.service └─4285 /home/igor/overpass/osm-3s_v0.7.54/bin/dispatcher --osm-base --db-dir=/home/igor/overpass/db
Querying The systemd Journal
Also, you can query the systemd journal:
journalctl -xe
(-x means log lines with be augmented with explanation texts from the message catalog, while -e means the end of the journal will be shown).
You can also display the journal entries just for the overpass service:
journalctl -xe -u overpass
To follow the journal, use
journalctl -xf -u overpass
Restarting The Overpass Service
f you made a mistake in the overpass.service configuration file, update the file and then stop and start the service:
sudo systemctl stop overpass sudo systemctl start overpass
Getting Information Status Information From The Running Dispatcher
You can get status info about from the running dispatcher:
~/overpass/osm-3s_v0.7.54/bin/dispatcher --status
For the list of all available command line parameters, execute
~/overpass/osm-3s_v0.7.54/bin/dispatcher --help
Configuring Diffs
We will register another systemd service that will perform the task of continually updating the database with the latest changes. First we need to prepare a systemd configuration file (called overpass-diff.service) for the dispatcher daemon:
sudo nano /lib/systemd/system/overpass-diff.service
Paste the following into the text editor once it opens and save the file. Note that you need to change the user account the service will be running as (the User directive) and the various parameters that point to the /home/igor/overpass directory:
[Unit] Description=Overpass diff init agent Requires=overpass.service After=overpass.service [Service] Type=simple User=igor ExecStart=/home/igor/overpass/osm-3s_v0.7.54/bin/fetch_osc_and_apply.sh http://planet.osm.org/replication/minute/ --meta=no [Install] WantedBy=multi-user.target
Also note we configured this service to be dependent on the main dispatcher overpass.service (Requires directive) and that it should be started after that service (After directive). You can use the same instructions as for the main service to register, start and troubleshoot it. You can also use
tail -f apply_osc_to_db.log
to follow what is going on with the service.
Setting Up The Web API
The goal of this section is to set up the web access to Overpass API using Apache web server. In my case I wanted to keep the default virtual host (on port 80) as it is, so I created a new virtual host on port 8091 just for Overpass API (as described below).
Preparing Apache Global Configuration
This step is highly dependent on your specific server/Apache configuration, but I'm writing it nonetheless: make sure you have the ServerName directive specified in the Apache's global configuration file (/etc/apache2/apache2.conf):
echo "ServerName localhost" | sudo tee /etc/apache2/conf-available/servername.conf && sudo a2enconf servername
(in my case, my server is called "jazz" and I will be using that name later in the virtual host configuration file).
Setting A Sensible Web Request Timeout Value
The default Timeout parameter value in the global configuration file (/etc/apache2/apache2.conf) of 500 seconds will probably not be enough for larger queries. If the query takes longer than that, you will get 504 Gateway Timeout error when the Apache aborts the request, so a longer timeout period is recommended.
Telling Apache To Listen To Port 8091
Since I will be using port 8091 for the Overpass API virtual host, we first need to tell Apache to listen to that port. This is accomplished by adding a new line in the listening ports configuration file (/etc/apache2/ports.conf):
Listen 8091
Opening Port 8091 In The Firewall
In order for the Overpass API website (and the web service) to be accessible remotely, we need to open the 8091 port on the firewall:
sudo iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 8091 -j ACCEPT
Enabling Required Apache Modules
We need to enable the CGI and ext_filter support in Apache2:
sudo a2enmod cgi sudo a2enmod ext_filter
Preparing Virtual Host Configuration
Now we need to prepare the configuration for the new virtual host, by copying the configuration of the default virtual host and using it as a template:
sudo cp /etc/apache2/sites-available/000-default.conf /etc/apache2/sites-available/overpass.conf
Now open the newly created overpass.conf file to edit it:
Paste the following into the text editor once it opens and save the file (note the various parameters point to the /home/igor/overpass directory, you need to change them for your own installation):
<VirtualHost *:8091>
ServerAdmin webmaster@localhost
ExtFilterDefine gzip mode=output cmd=/bin/gzip
DocumentRoot /home/igor/overpass/osm-3s_v0.7.54/html
ScriptAlias /api/ /home/igor/overpass/osm-3s_v0.7.54/cgi-bin/
<Directory "/home/igor/overpass/osm-3s_v0.7.54">
AllowOverride None
Options Indexes FollowSymLinks
Require all granted
</Directory>
<Directory "/home/igor/overpass/osm-3s_v0.7.54/cgi-bin/">
AllowOverride None
Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
Require all granted
</Directory>
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
Enabing The Virtual Host
Now we tell Apache to enable the virtual host we prepared:
sudo a2ensite overpass.conf
Reloading Apache Configuration
Now we tell Apache to reload the whole configuration so we can test Overpass API's website:
sudo service apache2 reload
Testing Overpass API's Website
In your browser, navigate to http://jazz:8091. It should display the entry page of Overpass API website with various links. You can navigate to the "Query and Convert Forms" link and enter a sample query there:
<query type="node"><bbox-query n="51.0" s="50.9" w="6.9" e="7.0"/><has-kv k="amenity" v="pub"/></query><print/>
Troubleshooting Overpass API Web Access
In case of problems, check Apache's error and access logs:
tail -f /var/log/apache2/error.log tail -f /var/log/apache2/access.log
You can also analyze Apache's configuration using the following command:
sudo /usr/sbin/apache2 -S
504 Gateway Timeout Error
If you are getting 504 Gateway Timeout error, you will probably have to increase the Timeout period in the Apache configuration, as specified in #Setting A Sensible Web Request Timeout Value section. After changing the setting, remember to reload the Apache configuration.