OpenFireMap/Installation
This page describes how to install the software you need for creating a map like openfiremap.org (see also OpenFireMap Wiki page).
Prerequisites
Hardware
A weak CPU or a virtual Internet server, will suffice if you limit the geographical region (e.g. France, Germany, Australia). For rendering a larger area, at least 8 GB RAM are recommended.
Operating System
We assume that you use Ubuntu >=20.04 as operating system. All of the following steps have been tested with version 20.04.
Prepare your System
This chapter describes the steps which are to be done once. After having run through this work, you will only need to update the OSM data for your map from time to time.
If you are not using a virtual machine solely for this purpose it is recommended to create a separate user. In this example, we decide to go on with the user root since the map is going to be the only task running on this virtual server.
Software Packages
At first, install all packages necessary to run your own web server. Additionally, you will need some other packages. Ensure the installation of all necessary packages by executing these commands:
apt update apt upgrade apt install nano unzip gcc zlib1g-dev apt install postgresql-12-postgis-3 postgresql-contrib apt install osm2pgsql apt install python3-mapnik mapnik-utils apt install python2.7-minimal apt install apache2 php php-pgsql libapache2-mod-php libjs-openlayers
At this point, it does not hurt to reboot the system to ensure all services have been started in proper order.
PostgreSQL Database
The GIS database must be initialized. Please follow these steps:
sudo -u postgres -i -H createuser -SdR hyuser createdb -E UTF8 -O hyuser hygis psql -d hygis -f /usr/share/postgresql/12/contrib/postgis-3.0/postgis.sql psql -d hygis -f /usr/share/postgresql/12/contrib/postgis-3.0/spatial_ref_sys.sql psql hygis -c "ALTER TABLE geography_columns OWNER TO hyuser" psql hygis -c "ALTER TABLE geometry_columns OWNER TO hyuser" psql hygis -c "ALTER TABLE spatial_ref_sys OWNER TO hyuser" exit
To enable easy database login by user hyuser we need to edit one of the database configuration files. In case you are running Ubuntu with a graphical interface, you could use a more comfortable editor, e.g. gedit instead of nano.
nano /etc/postgresql/12/main/pg_hba.conf
Near to the bottom of the file you will find these lines:
local all all peer host all all 127.0.0.1/32 md5 host all all ::1/128 md5
In these lines change the word "peer" and the two words "md5" each to "trust" and close the editor (for nano: Ctrl-O, Enter, Ctrl-X). Now reload the database configuration:
service postgresql reload
For a short test, login to the database by using the previously created database user hyuser:
psql hygis hyuser
Type \d to see a list with the two tables which we have created some steps above (geography_columns, geometry_columns, and spatial_ref_sys). Then logout with \q
If you encounter any problems, you may find a solution here: PostGIS.
Choose a Project Directory
Our example directory for downloading OSM data and generating the data base contents will be the subdirectory hy of the user root, "/root/hy", which can be abbreviated as "~/hy" if being logged in with this user. Let's create this directory and a directory for the rendering tool Mapnik:
mkdir /root/hy mkdir /root/hy/mapnik_hy
If you want to use a different directory, please create it and grant all access rights to your user – in case it is not root. You also will have to replace all cd /root/hy commands in the following examples with cd and the full path to your alternate directory.
Tools osmconvert and osmfilter
These tools are mainly used to accelerate the PostgreSQL import process. If we filter out all information we need and drop everything else, osm2pgsql will run faster. On top of this, database queries will be faster, so the rendering process will be accelerated as well. You can install these programs from Ubuntu repository. They are supplied by package osmctools. However, it is recommended to download and compile the source code because the binary may be out of date. To do this – downloading and compiling – from the command line, enter these commands:
cd /root/hy wget -O - http://m.m.i24.cc/osmconvert.c |cc -x c - -lz -O3 -o osmconvert wget -O - http://m.m.i24.cc/osmfilter.c |cc -x c - -O3 -o osmfilter
For further details on both tools refer to osmconvert and osmfilter.
Get Icons
Mapnik renderer will need certain icons to represent the objects we want to display on the map. You can create these icons by yourself or download sets of icons from various Internet sources. In this example we take the icon set from openfiremap.org:
cd /root/hy/mapnik_hy wget -r -l 1 -nd -A \*.png -P symbols http://openfiremap.de/f/symbols_hy/
Mapnik Renderer Initialization
For the rendering tool Mapnik, some initializations are to be done. Afterwards, a Mapnik style file needs to be created in folder /root/hy/mapnik_hy. You can create the file mapnik_hy.xml on your own or use one of the examples from the Internet. You also may download the OpenFireMap style file from here: openfiremap.de/f/mapnik_hy.xml
cd /root/hy/mapnik_hy/inc wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/datasource-settings.xml.inc.template wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/entities.xml.inc wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/fontset-settings.xml.inc.template wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-addressing.xml.inc wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-admin.xml.inc wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-aerialways.xml.inc wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-amenity-points.xml.inc wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-amenity-stations.xml.inc wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-amenity-symbols.xml.inc wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-buildings.xml.inc wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-citywall.xml.inc wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-ferry-routes.xml.inc wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-landcover.xml.inc wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-placenames.xml.inc wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-power.xml.inc wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-shapefiles.xml.inc wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-water.xml.inc wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layer-water_features.xml.inc wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/layers.xml.inc wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/inc/settings.xml.inc.template cd /root/hy/mapnik_hy mkdir /root/hy/mapnik_hy/world_boundaries wget -N https://svn.openstreetmap.org/applications/rendering/mapnik/generate_xml.py /usr/bin/python2.7 ./generate_xml.py --host localhost --user hyuser --dbname hygis --symbols ./symbols/ --world_boundaries ./world_boundaries/ --port '' --password '' wget -N http://openfiremap.de/f/mapnik_hy.xml
Create the Map
This chapter will show you how to create or update the tiles for your map. Tiles are small bitmaps which will be assembled to a whole map by the map browser. We assume that all tasks of the previous chapter have been completed successfully.
The process of map creation involves two steps – filling the database and rendering the map tiles.
Fill the Database
All information we need to create the map tiles must be written into our database. To do this, we need to develop a script which performs several tasks, step by step. We will do this step by step, entering every single command at the command line terminal. That makes it much easier to find errors.
Get the OSM Data File
It is not recommended to use the file of the entire planet. Please choose the file of an area you are interested in, in this example a part of Germany. The first task will be to download this file and to store it using .o5m file format.
cd /root/hy wget -N http://download.geofabrik.de/europe/germany/bayern/mittelfranken-latest.osm.pbf -O - |./osmconvert - -o=a_hy.o5m
Filter the OSM Data File
We need to do a hierarchical filtering because there will be nodes in ways, ways in relations and relations in other relations. For performance reasons a pre-filtered file will be used for interrelational filtering. In this example we decided to filter public-transport specific information because we want to create a public transport map. The filtering criteria need to be specified in a file. This can be done via commandline like this:
cd /root/hy cat <<eof >toolchain_hy.filter --keep= emergency=fire_hydrant =water_tank =suction_point =fire_water_pond amenity=fire_hydrant =fire_station --keep-tags=all emergency= amenity= name= ref= fire_hydrant:type= fire_hydrant:diameter= fire_hydrant:pressure= fire_hydrant:count= fire_hydrant:position= water_tank:volume= type= eof
Then the OSM raw data can be filtered:
./osmfilter a_hy.o5m --parameter-file=toolchain_hy.filter -o=f_hy.o5m
Some seconds or minutes later – depending on the size of the chosen area – we will get the file gis_hy.o5m, containing only that information we need.
Note: We are using .o5m file format, because this will accelerate the filter process. For further information see osmconvert and .o5m.
Transfer the Data to Postgres Database
Now we can transfer the OSM data from the file f_hy.o5m into the Postgres database. The program osm2pgsql will do this job. To fit the needs of our specialised map, we need to create our own osm2pgsql style file first. This can be done with an editor or by executing these commands:
cd /root/hy cat <<eof >osm2pgsql_hy.style node,way emergency text polygon node,way amenity text polygon node,way name text linear node,way ref text linear node,way fire_hydrant:type text linear node,way fire_hydrant:diameter text linear node,way fire_hydrant:pressure text linear node,way fire_hydrant:count text linear node,way fire_hydrant:position text linear node,way water_tank:volume text linear eof
Now you can start the data import:
osm2pgsql -s -C 1500 -d hygis -U hyuser -S osm2pgsql_hy.style -c f_hy.o5m -e 18-18 -o dirty_tiles_hy
Because the .o5m file had been filtered in the previous step, this transfer should take only a few minutes. If you have more main memory to spend, you can increase the number of available MB increasing the parameter -C 1500.
If you encounter any problems, you may find a solution here: osm2pgsql.
Getting a List of all Dirty Tiles
Map tiles which are not up-to-date are referred to as dirty tiles. During map data import the program osm2pgsql generates a list of all dirty tiles. Unfortunately the program does not provide us with the names of all affected tiles because this list is stripped of all redundant information. If there is, for example, the tile 3/0/1 (zoom, x, y) in the list, the also affected tiles 2/0/0 and 1/0/0 will be omitted. Furthermore, a list entry 3/0/1 means that all four tiles of the next zoom level have also been omitted although affected: 4/0/2, 4/0/3, 4/1/2, and 4/1/3.
This is an effective way to reduce the dirty-tiles list length but we still need a list of all effected tiles. For this reason, the dirty tiles list must be expanded accordingly. This can be done with a small C program which needs to be added to our toolchain:
cd /root/hy
cat <<eof |cc -x c - -O3 -o dtexpand
// dtexpand 2017-03-10 18:00
//
// expands the dirty-tiles list as provided by osm2pgsql
// example: dtexpand 4 17 <dirty_tiles >dirty_tiles_ex
// (c) 2017 Markus Weber, Nuernberg, License LGPLv3
#include <ctype.h>
#include <inttypes.h>
#include <string.h>
#include <stdlib.h>
#include <stdio.h>
int main(int argc,char** argv) {
int fieldedge[20];
uint8_t* field[20],*fieldend[20];
int zoom_min,zoom_max,zoom_sub;
int z,x,y,zz,xx,yy,xd,yd,q; uint64_t u;
uint8_t b; uint8_t* bp,*bpa,*bpe;
char line[40]; char* sp;
typedef struct {int zs,tx_min,tx_max,ty_min,ty_max;} sub_t;
sub_t* sub,*subp; int subn;
if(argc<3 || (argc-3)%5!=0) { fprintf(stderr,
"Usage: dtexpand ZOOM_MIN ZOOM_MAX <DT_IN >DT_OUT\n"
"Optional value sets each: ZOOM_SUB TX_MIN TX_MAX TY_MIN TY_MAX\n"
" ZOOM_SUB: max zoom level for blindly adding subtiles\n"
" TX...TY: concerning tile range at ZOOM_MAX level\n"
); return 1; }
zoom_min= atoi(argv[1]); zoom_max= atoi(argv[2]);
subn= (argc-3)/5;
sub= (sub_t*)malloc(sizeof(sub_t)*(subn+1));
argv+= 2; subp= sub;
for(z= 0;z<subn;z++) { // for each subtile value set
subp->zs= atoi(*++argv);
subp->tx_min= atoi(*++argv); subp->tx_max= atoi(*++argv);
subp->ty_min= atoi(*++argv); subp->ty_max= atoi(*++argv);
if(zoom_min<0 || zoom_min>19 || zoom_max<zoom_min || zoom_max>19 ||
subp->zs<=zoom_max || subp->zs>19) { fprintf(stderr,
"Unsupported zoom value(s).\n"); return 2; }
subp++; } // for each subtile value set
subp->zs= 0;
for(z= zoom_max-1;z>=zoom_min;z--) { // for each zoom level
fieldedge[z]= u= UINT32_C(1)<<z; u= u*u+7>>3;
if((field[z]= (uint8_t*) malloc(u))==NULL) { fprintf(stderr,
"Not enough main memory.\n"); return 3; }
memset(field[z],0,u);
fieldend[z]= field[z]+u;
} // for each zoom level
while(fgets(line,sizeof(line),stdin)!=NULL) { // for each input line
sp= line;
z= 0; while(isdigit(*sp)) z= z*10+*sp++-'0'; if(*sp=='/') sp++;
x= 0; while(isdigit(*sp)) x= x*10+*sp++-'0'; if(*sp=='/') sp++;
y= 0; while(isdigit(*sp)) y= y*10+*sp++-'0';
if(z<zoom_min || z>zoom_max)
continue;
zz= z-1; xx= x>>1; yy= y>>1;
while(zz>=zoom_min) { // for all lower zoom levels
u= xx; u*= fieldedge[zz]; u+= yy; bp= &field[zz][u/8];
if(bp<fieldend[zz]) *bp|= 1<<(u&7); // (preventing overflow)
zz--; xx>>= 1; yy>>= 1;
} // for all lower zoom levels
q= 1;
do { // for this and all higher zoom levels
for(xd= 0;xd<q;xd++) for(yd= 0;yd<q;yd++) { // all these tiles
xx= x+xd; yy= y+yd;
if(z<zoom_max) {
u= xx; u*= fieldedge[z]; u+= yy; bp= &field[z][u/8];
if(bp<fieldend[z]) *bp|= 1<<(u&7); // (preventing overflow)
}
else { // at zoom_max level
printf("%i/%i/%i\n",z,xx,yy);
// determine relevant subtile zoom level
zoom_sub= zoom_max;
subp= sub;
while(subp->zs!=0) { // for each subtile value set
if(subp->zs>zoom_sub &&
xx>=subp->tx_min && xx<=subp->tx_max &&
yy>=subp->ty_min && yy<=subp->ty_max) zoom_sub= subp->zs;
subp++; } // for each subtile value set
// write subtiles
int xxd,yyd,qq;
zz= z; qq= 1;
while(zz<zoom_sub) { // for each subtile level
zz++; xx<<= 1; yy<<= 1; qq<<= 1;
for(xxd= 0;xxd<qq;xxd++) for(yyd= 0;yyd<qq;yyd++)
printf("%i/%i/%i\n",zz,xx+xxd,yy+yyd);
} // for each subtile level
} // at zoom_max level
} // all these tiles
z++; x<<= 1; y<<= 1; q<<= 1;
} while(z<=zoom_max); // for this and all higher zoom levels
} // for each input line
for(z= zoom_max-1;z>=zoom_min;z--) { // for each zoom level
bp= bpa= field[z]; bpe= fieldend[z];
do { // for each byte in bitfield
b= *bp;
if(b) { // at least one bit in byte is set
int be= fieldedge[z];
for(int bi=0;bi<8;bi++) { // for each bit in byte
if(b&1) { // bit is set
u= (bp-bpa)*8+bi; x= u/be; y= u&be-1;
printf("%i/%i/%i\n",z,x,y);
} // bit is set
b>>= 1;
} // for each bit in byte
} // at least one bit in byte is set
} while(++bp<bpe); // for each byte in bitfield
free(field[z]);
} // for each zoom level
free(sub); return 0;
} // main()
eof
Besides its main function this program can also be used to add subtile names for certain regions to the dirty-tiles list. If you want to create a global map up to zoom level 17, for example, and having a small region rendered up to zoom level 18, you may later add this subtile zoom level as well as the region's level-17 tile range to the command line:
./dtexpand 4 17 18 69000 70000 44000 45000
If there are two or more of such regions, just enter all of their 5-value sets. For example:
./dtexpand 4 17 19 80000 81000 48000 49000 18 69000 70000 44000 45000
At this point we do not need to care about expanding the list of dirty tiles since that will be dealt with by the toolchain script which is going to be introduced some sections below.
Rendering Test
At first, we should test if the renderer works as expected. Thereby, the nik4 program will help us. Let's pick an area for which we have already imported GIS data – in this example: Nürnberg, Germany – generate a test image and examine it with the viewer (here Eye of Gnome).
cd /root/hy/mapnik_hy wget -N https://raw.githubusercontent.com/Zverik/Nik4/master/nik4.py /usr/bin/python3 nik4.py -c 11 49.5 -z 12 -d 800 400 -f png256 mapnik_hy.xml image.png eog image.png &
Render the Tiles
If the test-run has been successful, the next step should be to generate map tiles. For this purpose, we use the scripts render_hy.py and toolchain_hy.sh which must be created first:
cd /root/hy nano render_hy.py
Enter the following lines:
#!/usr/bin/python
# render.py 2020-12-25 15:50
# reads dirty-tiles file and renders each tile of this list;
# afterwards, deletes the rendered dirty-tiles file;
# call: LAY=hy DTF=dirty_tiles_hy render.py
# parallel call: LAY=hy DTF=dirty_tiles_hy PID=123 render.py
# adaptations: Markus Weber, Nuernberg
from math import pi,cos,sin,log,exp,atan
from subprocess import call
import sys, os
import mapnik
import threading
import shutil
DEG_TO_RAD = pi/180
RAD_TO_DEG = 180/pi
def minmax (a,b,c):
a = max(a,b)
a = min(a,c)
return a
class GoogleProjection:
def __init__(self,levels=18):
self.Bc = []
self.Cc = []
self.zc = []
self.Ac = []
c = 256
for d in range(0,levels):
e = c/2;
self.Bc.append(c/360.0)
self.Cc.append(c/(2 * pi))
self.zc.append((e,e))
self.Ac.append(c)
c *= 2
def fromLLtoPixel(self,ll,zoom):
d = self.zc[zoom]
e = round(d[0] + ll[0] * self.Bc[zoom])
f = minmax(sin(DEG_TO_RAD * ll[1]),-0.9999,0.9999)
g = round(d[1] + 0.5*log((1+f)/(1-f))*-self.Cc[zoom])
return (e,g)
def fromPixelToLL(self,px,zoom):
e = self.zc[zoom]
f = (px[0] - e[0])/self.Bc[zoom]
g = (px[1] - e[1])/-self.Cc[zoom]
h = RAD_TO_DEG * ( 2 * atan(exp(g)) - 0.5 * pi)
return (f,h)
if __name__ == "__main__":
# read environment variables
lay = os.environ['LAY']
dt_file_name = os.environ['DTF']
try:
temp_file = "/dev/shm/tile" + os.environ['PID']
except:
temp_file = "/dev/shm/tile"
# do some global initialization
os.chdir("mapnik_"+lay+"/")
#sys.path.append("mapnik")
print("render.py " + dt_file_name + ": started.")
mapfile = "mapnik_" + lay + ".xml"
###############################################################
tile_dir = "/var/www/html/" + lay + "tiles_new/"
###############################################################
max_zoom = 18
tile_count = 0
empty_tile_count = 0
# create tile directory and all possible zoom directories
if not os.path.isdir(tile_dir):
os.mkdir(tile_dir)
for z in range(0, max_zoom+1):
d = tile_dir + "%s" % z + "/"
if not os.path.isdir(d):
os.mkdir(d)
# open the dirty-tiles file
try:
dt_file = open("../" + dt_file_name, "r")
except:
dt_file = None
print("render.py: dt file not found: "+dt_file_name)
if (dt_file != None):
# do some Mapnik initialization
mm = mapnik.Map(256, 256)
mapnik.load_map(mm, mapfile, True)
# Obtain <Map> projection
prj = mapnik.Projection(mm.srs)
# Projects between tile pixel co-ordinates and LatLong (EPSG:4326)
tileproj = GoogleProjection(max_zoom + 1)
# process the dirty-tiles file
while True:
# read a line of the dirty-tiles file
line = dt_file.readline()
if (line == ""):
break
line_part = line.strip().split('/')
# create x coordinate's directory - if necessary
d = tile_dir + line_part[0] + "/" + line_part[1] + "/"
if not os.path.isdir(d):
os.mkdir(d)
# get parameters of the line
z = int(line_part[0])
x = int(line_part[1])
y = int(line_part[2])
tile_name= tile_dir + line[:-1] + ".png"
# render this tile - start
# Calculate pixel positions of bottom-left & top-right
p0 = (x * 256, (y + 1) * 256)
p1 = ((x + 1) * 256, y * 256)
# Convert to LatLong (EPSG:4326)
l0 = tileproj.fromPixelToLL(p0, z);
l1 = tileproj.fromPixelToLL(p1, z);
# Convert to map projection (e.g. mercator co-ords EPSG:900913)
c0 = prj.forward(mapnik.Coord(l0[0], l0[1]))
c1 = prj.forward(mapnik.Coord(l1[0], l1[1]))
# Bounding box for the tile
if hasattr(mapnik,'mapnik_version') and mapnik.mapnik_version() >= 800:
bbox = mapnik.Box2d(c0.x,c0.y, c1.x,c1.y)
else:
bbox = mapnik.Envelope(c0.x,c0.y, c1.x,c1.y)
render_size = 256
mm.resize(render_size, render_size)
mm.zoom_to_box(bbox)
mm.buffer_size = 256
# (buffer size was 128)
# render image with default Agg renderer
im = mapnik.Image(render_size, render_size)
mapnik.render(mm, im)
im.save(temp_file, 'png') # changed from 'png256' to 'png', 2012-03-21
tile_count = tile_count + 1
# render this tile - end
# copy this tile to tile tree
l= os.stat(temp_file)[6]
if l>116:
shutil.copyfile(temp_file,tile_name)
else:
try:
os.unlink(tile_name)
except:
l= l # (file did not exist)
empty_tile_count = empty_tile_count + 1
# close and delete the dirty-tiles file
dt_file.close()
try:
os.unlink("../" + dt_file_name)
except:
l= l # (file did not exist)
# print some statistics
print("render.py " + dt_file_name + ":", tile_count, "tiles (" + "%s" % empty_tile_count + " empty).")Now open the toolchain file:
nano toolchain_hy.sh
Enter the following lines:
#!/bin/bash
# toolchain.sh
# This script cares about creating a thematical map.
# (c) Markus Weber, 2020-12-26 17:00
# License: AGPL V3
LAY="hy"
PLANETURL=http://download.geofabrik.de/europe/germany/bayern/mittelfranken-latest.osm.pbf
PLANETMINSIZE=60000000 # minimum size of OSM data file in .o5m format
TILEMINSIZE=130000 # minimum size of tile directory in blocks
#BORDERS="-B=bayern.poly"
MAXPROCESS=2 # maximum number of concurrent processes for rendering
OSM2PGSQLPARAM="-s -C 1500 -d "$LAY"gis -U "$LAY"user -S osm2pgsql_"$LAY".style"
# main parameters to be passed to osm2pgsql
# do some initializations
PLANETFILE="a_"$LAY".o5m"
LOG="tc_"$LAY".log"
DTLOG="dt_"$LAY".log"
RUNNING="toolchain_running_"$LAY".txt"
ENDED="toolchain_ended_"$LAY".txt"
FILTER="f_"$LAY".o5m"
DTDIR="dt_"$LAY"/"
DTFILE="dirty_tiles_"$LAY
PROCN=1000 # rendering-process number (range 1000..1999)
mkdir $DTDIR 2>/dev/null
# enter working directory and start logging
if [ "0"$LAY = "0" ]; then exit 1; fi
cd /root/hy
echo >>$LOG
echo $(date)" toolchain started." >>$LOG
mkdir "dt_"$LAY"/" 2>/dev/null
# publish that this script is now running
rm -f $ENDED
echo -e "The toolchain script has been started at "$(date)"\n"\
"and is currently running. To terminate it, delete this file and\n"\
"wait until the file \""$ENDED"\" has been created.\n"\
"This may take some minutes." \
>$RUNNING
# clean up previous Mapnik processes
killall "mapnik_"$LAY".py" 2>/dev/null
while [ $(ls $DTDIR""at* 2>/dev/null |wc -l) -gt 0 ]; do
# there is at least one incompleted rendering process
AT=$(ls -1 $DTDIR""at* 2>/dev/null|head -1) # tile list
echo $(date)" Cleaning up incomplete rendering: "$AT >>$LOG
DT=${AT:0:6}d${AT:7:99} # new name of the tile list
mv $AT $DT # rename this tile list file;
# now the tile is marked as 'to be rendered'
done
# delete old tile files
echo $(date)" Deleting old tile files." >>$LOG
find /var/www/html/"$LAY"tiles_old/ -type f -delete 2>/dev/null
rm -rf /var/www/html/"$LAY"tiles_old 2>>$LOG
# download and process planet file - if necessary
if [ "0"$(stat --print %s $PLANETFILE 2>/dev/null) -lt $PLANETMINSIZE ]; then
echo $(date)" Missing file $PLANETFILE, downloading it." >>$LOG
rm -f $PLANETFILE
wget -nv $PLANETURL -O - 2>>$LOG |./osmconvert - \
$BORDERS --drop-author -o=$PLANETFILE 2>>$LOG
if [ "0"$(stat --print %s $PLANETFILE 2>/dev/null) -lt $PLANETMINSIZE ]; then
echo $(date)" toolchain Error: could not download"\
$PLANETURL >>$LOG
exit 1
fi
rm -f $DTFILE $DTDIR""*
echo $(date)" Filtering the downloaded planet file." >>$LOG
./osmfilter $PLANETFILE --parameter-file=toolchain_"$LAY".filter \
-o=$FILTER 2>>$LOG # filter the planet file
echo $(date)" Writing filtered data into the database." >>$LOG
osm2pgsql $OSM2PGSQLPARAM -c $FILTER -e 18-18 -o $DTFILE >/dev/null 2>&1
# enter filtered planet data into the database
echo $(date)" All tiles need to be rerendered." >>$LOG
# prepare tiles directory
D=/var/www/html/"$LAY"tiles_new
mkdir $D $D/0 $D/1 $D/2 $D/3 $D/4 $D/5 $D/6 $D/7 $D/8 $D/9 $D/10 \
$D/11 $D/12 $D/13 $D/14 $D/15 $D/16 $D/17 $D/18 2>/dev/null
fi
# main loop
while [ -e $RUNNING ]; do
echo $(date)" Processing main loop." >>$LOG
# limit log file size
if [ "0"$(stat --print %s $LOG 2>/dev/null) -gt 1500000 ]; then
echo $(date)" Reducing logfile size." >>$LOG
mv $LOG $LOG"_temp"
tail -c +1000000 $LOG"_temp" |tail -n +2 >$LOG
rm -f $LOG"_temp"
fi
#care about entries in dirty-tile list
MAINTCOUNT=0
while [ $(ls $DTFILE $DTDIR""?t* 2>/dev/null |wc -l) -gt 0 -a \
-e $RUNNING ]; do
# while still tiles to render
# clean-up failed rendering tasks (about every 20 minutes)
MAINTCOUNT=$(($MAINTCOUNT + 1))
if [ MAINTCOUNT -gt 200 ]; then
T=$(find d_t_"$LAY" -name "at*" -mmin +100|head -1|tail -c +7)
if [ "1"$T -gt "1" ] ; then
mv -n d_t_"$LAY"/at$T d_t_"$LAY"/dt$T 2>/dev/null
echo $(date)" Restarted rendering: "$T >>$LOG
fi
MAINTCOUNT=0
fi
# start as much render processes as allowed
while [ $(ls $DTDIR""dt* 2>/dev/null |wc -l) -gt 0 -a \
$(ls -1 $DTDIR""at* 2>/dev/null |wc -l) -lt $MAXPROCESS ]; do
# while dirty tiles in list AND process slot(s) available
DT=$(ls -1 $DTDIR""dt* |head -1) # tile list
AT=${DT:0:6}a${DT:7:99} # new name of the tile list
touch -c $DT # remember rendering start time
mv $DT $AT # rename this tile list file;
# this is our way to mark a tile list as 'being rendered now'
#echo $(date)" Rendering "$DT >>$LOG
PROCN=$(($PROCN + 1))
if [ $PROCN -gt 1999 ]; then
PROCN=1000; # (force range to 1000..1999)
fi
N=${PROCN:1:99} # get last 3 digits
LAY=$LAY DTF=$AT PID=$N nohup python3 render_"$LAY".py >/dev/null 2>&1 &
# render every tile in list
echo $(date)" Now rendering:"\
$(ls -m $DTDIR""at* 2>/dev/null|tr -d $DTDIR"a ") \
>>$LOG
done
# determine if we have rendered all tiles of all lists
if (ls $DTDIR""?t* >/dev/null 2>&1); then # still tiles to render
sleep 6 # wait some seconds
continue # care about rendering again
fi
# here: we have rendered all tiles of all lists
# care about dirty-tiles master list and split it into parts
if (ls $DTFILE >/dev/null 2>&1); then
# there is a dirty-tiles master list
echo "=== "$(date) >>$DTLOG
cat $DTFILE >>$DTLOG # add list to dirty-tiles log
echo $(date)" Expanding \"dirty_tiles\" file." >>$LOG
./dtexpand 11 18 <$DTFILE >$DTFILE"_ex" 2>/dev/null
mv -f $DTFILE"_ex" $DTFILE 2>/dev/null
echo $(date)" Splitting dirty-tiles list" \
"("$(cat $DTFILE |wc -l)" tiles)" >>$LOG
split -l 1000 -d -a 6 $DTFILE $DTDIR""dt
echo "+++ "$(date) >>$DTLOG
cat $DTFILE >>$DTLOG # add list to dirty-tiles log
rm -f $DTFILE
# limit dirty-tiles log file size
if [ "0"$(stat --print %s $DTLOG 2>/dev/null) -gt 750000000 ]; then
echo $(date)" Reducing dirty-tiles logfile size." >>$LOG
mv $DTLOG $DTLOG"_temp"
tail -c +500000000 $DTLOG"_temp" |tail -n +2 >$DTLOG
rm -f $DTLOG"_temp"
fi
fi
done # while still tiles to render
if [ ! -e $RUNNING ]; then # script shall be terminated
continue; # exit the main loop via while statement
fi
# here: all tiles have been rendered
# check if there is a plausible amount of tiles
if [ "0"$(du -s /var/www/html/"$LAY"tiles_new 2>/dev/null|cut -f 1 2>/dev/null) \
-lt $TILEMINSIZE ]; then
echo $(date)" toolchain Error: implausible amount of tiles" >>$LOG
exit 1
fi
# move tile directory
echo $(date)" Moving tile directory." >>$LOG
ls -lL $PLANETFILE $FILTER >>$LOG
rm -f $PLANETFILE $FILTER
mv -f /var/www/html/"$LAY"tiles /var/www/html/"$LAY"tiles_old 2>/dev/null
mv -f /var/www/html/"$LAY"tiles_new /var/www/html/"$LAY"tiles 2>/dev/null
break
done # main loop
# wait until all rendering processes have terminated
while (ls $DTDIR""at* 2>/dev/null) ; do sleep 30; done
# publish that this script has ended
rm -f $RUNNING
echo "The toolchain script ended at "$(date)"." \
>$ENDED
echo -e $(date)" Toolchain ended.\n" >>$LOGMake the shell script executable:
chmod +x toolchain_hy.sh
Now start the toolchain and wait until it is completed. Before this you may want to change some parameters in the toolchain script: the number of rendering processes, for example.
cd /root/hy ./toolchain_hy.sh
Depending on the size of the region you have downloaded this step may take a few minutes up to several hours. You can watch the rendering process(es) by invoking the process monitor top (from a separate terminal window):
top
Inspect the logfile tc_hy.log for any possible errors which might have occurred. If everything went fine, it is time to automatise the rebuild process. This can be done via crontab:
crontab -e
Enter this line:
22 5 3 * * /root/hy/toolchain_hy.sh
Now the map will be rebuilt automatically every month (at 05:22 on day 3). Please be aware that all OSM data will be downloaded from the location you have chosen above and therefore lead to some effort on the side of that company. Please refer to their terms for using the service. If you are building this map for commercial purposes, please consider to make a donation – no matter if it is legally required or not.
Build the Website
The web server Apache will expect our web content at "/var/www/html/". To display, move and zoom the map on the screen a specialized framework will be needed: openlayers. For this, some additional files must be copied resp. downloaded:
cd /var/www/html cp /usr/share/javascript/openlayers/OpenLayers.js . cp /usr/share/openlayers/theme/default/style.css . mkdir img cp /usr/share/openlayers/img/* img/ wget -N https://www.openstreetmap.org/openlayers/OpenStreetMap.js
Now open the existing example index file and replace its contents by the following text. You can use gedit or nano editor for this task: nano /var/www/html/index.html
<!DOCTYPE HTML>
<html> <head>
<title>OpenFireMap</title>
<title>Fire Hydrants</title>
<link rel="stylesheet" href="style.css" type="text/css">
<style type="text/css">
html, body, #map {
width: 100%;
height: 100%;
margin: 0;
}
.olImageLoadError {
display: none !important;
}
</style>
<script src="OpenStreetMap.js"></script>
<script src="OpenLayers.js"></script>
<script>
function init() {
OpenLayers.Util.onImageLoadError= function() {
this.src= "emptytile.png"; };
map= new OpenLayers.Map("map", { controls:[
new OpenLayers.Control.Navigation(),
new OpenLayers.Control.PanZoomBar(),
new OpenLayers.Control.LayerSwitcher(),
new OpenLayers.Control.Permalink(),
new OpenLayers.Control.ScaleLine(),
new OpenLayers.Control.Permalink('permalink'),
new OpenLayers.Control.MousePosition(),
new OpenLayers.Control.Attribution()],
maxResolution: 156543.0399,numZoomLevels: 18, units: 'm',
projection: new OpenLayers.Projection("EPSG:900913"),
displayProjection: new OpenLayers.Projection("EPSG:4326"),
restrictedExtent: new OpenLayers.Bounds(550000,5700000,2000000,7450000)
} );
map.addLayer(new OpenLayers.Layer.OSM("OSM"));
map.addLayer(new OpenLayers.Layer.OSM("No Background","img/blank.png"));
map.addLayer(new OpenLayers.Layer.OSM("Fire hydrants","hytiles/${z}/${x}/${y}.png",
{ maxZoomLevel: 18, numZoomLevels: 19, alpha: true, isBaseLayer: false }));
if(!map.getCenter()){
map.setCenter(new OpenLayers.LonLat(11.075,49.455).transform(
new OpenLayers.Projection("EPSG:4326"),map.getProjectionObject()),11);
}
}
</script>
</head><body onload="init();">
<div style="width:100%; height:100%;" id="map"></div><br>
<div style="border:1px solid black; padding:10px;">
This map shows those fire hydrants which have been entered into the OpenStreetMap database. <a href="http://openstreetmap.org/copyright" target="_blank">(c) OpenStreetMap contributors</a>
<br><br>
Further information can be found at OpenStreetMap Wiki in
<a href="http://wiki.openstreetmap.org/wiki/OpenFireMap" target="_blank">English</a>,
<a href="http://wiki.openstreetmap.org/wiki/DE:OpenFireMap" target="_blank">German</a> and
<a href="http://wiki.openstreetmap.org/wiki/Pl:OpenFireMap" target="_blank">Polish</a><br>
</div>
</body></html>Further information about OpenLayers can be found here.
Test your new Map
Open a web browser and try to access your new website. If your Internet browser is on the same computer as the Apache server, type localhost or 127.0.0.1 as URL. If you did not render the complete map yet but have rendered that single tile from the rendering test example above, you will find it here: 127.0.0.1/?zoom=10&lat=49.5&lon=11&layers=B0000T
Please be aware that this map is an overlay map. That means in this case, the background tiles are fetched directly from openstreetmap.org map server, and just the foreground tiles – the OpenFireMap layer – comes from your server.
Benchmarks
Example setup: weak virtual server, two cores, OSM data from DACH region (Germany, Austria, Switzerland)
- 10 minutes downloading and converting the OSM data file
- 2 minutes filtering the file
- 20 seconds writing the filtered data into the database
- 2 hours rendering all tiles of the OpenFireMap overlay (1.2 Mio. tiles)
Troubleshooting
- Too many links
If this message occurs during rendering, you most likely are using a file system which supports only a limited number of subdirectories (e.g. ext2 or ext3). Please upgrade to ext4. The migration from ext3 to ext4 should be possible without any loss of data. Please refer to the users guide of your operating system.