Fugro/Setup

The Fugro aerial imagery layer is based on a bunch of data and software. At the moment the layer is maintained by Peter Brodersen.

This page describes our general server setup. The setup is Linux based; Debian Stable (Lenny).

Data source

The data source is 662 .ecw files from Fugro Aerial Mapping A/S. The files span from few MB to 235 MB. The total size is 77.4 GB.

Every file covers a quadrant of 10km x 10km in EPSG25832 projection (Euref89/ETRS89). The files are named after their position, e.g. 10km_636_59.ecw - where 636 og 59 is the Northing and Easting (* 10 km) values.

Index

All files are indexed in a Shapefile using gdalindex:

gdaltindex -tileindex LOCATION -write_absolute_path fugroindex.shp 10km_ecw_e89/*.ecw

This updates (and creates if necessary) the file fugroindex.shp which we point Mapserver at.

Mapserver

The primary tool is Mapserver which is a CGI application. At default the GDAL library (that mapserver is linked to) does not support ECW, which we then have to compile.

For Debian Stable (Lenny) we have fetched the following resources with git:

git clone http://rivendell.lovergine.com/pubgit/ecw.git/
git clone http://rivendell.lovergine.com/pubgit/libgdal-ecw.git/

First .deb files are built from ecw just by running debuild. After they are installed a .deb is built from libgdal-ecw in the same way, and is installed.

You may have trouble building the libgdal-ecw package. If so, you can build and install it manually (it is only a single file) using the following:

sh configure --with-ecw=/usr --with-autoload=/usr/lib/gdal16plugins --with-include=/usr/include/ecw
make
sudo mkdir -p /usr/lib/gdal16plugins
sudo cp gdal_ECW_JP2ECW.so /usr/lib/gdal16plugins

Now there is support for ECW files in libgdal. This can be confirmed with:

gdalinfo --formats|grep -i ecw

.. which would output something like:

ECW (rw): ERMapper Compressed Wavelets
JP2ECW (rw+): ERMapper JPEG2000

Furthermore, mapserver from Debian Stable (Lenny) is an old version that does not support tilemode. If you run a newer version than Lenny (or perhaps another distribution) with a newer mapserver, it should work now. Otherwise we would have to compile a newer version af mapserver ourselves, just by downloading the mapserver source, and configure and make. The produced mapserv file can be moved to /usr/lib/cgi-bin/ or another adequate place for script execution.

In our case mapserver was configured with these options:

./configure --with-proj --with-gdal --with-agg --with-jpeg

Mapfiles

Mapfiles are configuration files for Mapserver, which describes the options of what is to be outputted. These refer to the fugroindex.shp shapefile.

Ideally one could do with just one mapfile with different layers configured.

Photos

fugro.map with the layer "fugro" is used to output tiles:

MAP
    IMAGETYPE           jpeg
    EXTENT              470000 6070000 730000 6250000
    SIZE                400 300
    IMAGECOLOR          255 255 255

    PROJECTION
        "init=epsg:25832"
    END

    LAYER
        NAME            "fugro"
        STATUS          ON
        TILEINDEX       "/home/username/web/fugro/mapserv/fugroindex.shp"
        TILEITEM        "LOCATION"
        TYPE            RASTER
    END
END

Overview

overview.map is an overlay that shows which parts the different .ecw files covers. It can also be used if an area contains error, but the layer itself might not be that useful inside an editor. The overview also helped prioritizing the queue of files for upload, before they were all online.

The output can be seen at http://tile.openstreetmap.dk/?layers=BFT

MAP
    IMAGETYPE           transpng
    EXTENT              440000.000000 6040000.000000 900000.000000 6410000.000000
    SIZE                3000 1800
    IMAGECOLOR          255 255 255
    SHAPEPATH           "/home/username/web/fugro/mapserv/10km_ecw_e89"
    FONTSET             "/home/username/web/fugro/mapserv/fonts.list"

    PROJECTION
        "init=epsg:25832"
    END

    OUTPUTFORMAT
        NAME            "transpng"
        DRIVER          AGG/PNG
        MIMETYPE        "image/png"
        IMAGEMODE       RGBA
        TRANSPARENT     ON
        EXTENSION       "png"
    END

    LAYER
        NAME            "denmark"
        STATUS          ON
        DATA            ortooversigt
        TYPE            POLYGON
        LABELITEM       "NAVN"

        CLASS
            NAME                "Denmark"
            STYLE
                COLOR           232 232 232
                OUTLINECOLOR    32  32  32
            END
            LABEL
                COLOR           255 0 0
                SHADOWCOLOR     255 255 255
                SHADOWSIZE      0 0
                TYPE            TRUETYPE
                FONT            DejaVuSans
                SIZE            9
                ANTIALIAS       TRUE
                POSITION        CC
                PARTIALS        FALSE
                MINDISTANCE     300
                BUFFER          4
                                WRAP "_"
            END
        END
    END

    LAYER
        NAME            "fugro"
        STATUS          ON
        TILEINDEX       "/home/username/web/fugro/mapserv/fugroindex.shp"
        TILEITEM        "LOCATION"
        TYPE            RASTER
    END

END

The overview uses ortooversigt.shp, which was shipped along the imagery files and is found in the same folder as the imagery files (10km_ecw_e89).

fonts.list is just a reference to a TrueType file and simply contains:

DejaVuSans /usr/share/fonts/truetype/ttf-dejavu/DejaVuSans.ttf

Apache

The server already runs Apache 2.2. A virtual host for the tileserver purpose has been created with the appropriate configuration.

The direct URL to a tile is e.g.:

/cgi-bin/mapserv?map=/home/username/web/fugro/mapserv/fugro.map&layers=fugro&mode=tile&tilemode=gmap&tile=17472+10400+15

This includes a reference to the mapfile fugro.map, the name of the layer, that we want a tile as output, that the coordinates are in gmap format and finally the coordinate for our tile (x,y,z).

As most of these parameters aren't relevant for the end user, and that we would like the option of changing the configuration and renaming files without changing the URLs we choose to rewrite the URLs. This is also for the sake of more simple URLs in the editors' configuration.

Mapserver requires a lot of resources, both with regard to memory and CPU. An obvious optimization is caching of generated tiles. This can be done in several ways (Varnish, Nginx, TileCache and so on). We have chosen Apache's own mod_cache to avoid introducing more layers of software, and to be able to control the caching at the virtual host levels, regardless of other existing virtual hosts at the server.

Now and then mapserver fails at the generation of a tile, usually if the server runs out of memory in the process. As requests are very intensive the memory usage oscillates a lot with several simultaneous users. If maserver crashes under the way a tile will not be generated (and therefore fortunately won't end up as a "dead" tile in the cache). The end user can usually just reload or zoom in and out to regenerate the tile.

RewriteRule

We have the following rules in a .htaccess file, which is located in DocumentRoot for the virtual host:

RewriteEngine On
# Aerial Imagery and File Reference
RewriteRule ^fugro2005//?([0-9]+)/([0-9]+)/([0-9]+)\.(png|jpe?g)$ /cgi-bin/mapserv?map=/home/username/web/fugro/mapserv/fugro.map&layers=fugro&mode=tile&tilemode=gmap&tile=$2+$3+$1 [PT,L]
RewriteRule ^fugro2005/overview/([0-9]+)/([0-9]+)/([0-9]+)\.(png|jpe?g)$ /cgi-bin/mapserv?map=/home/username/web/fugro/mapserv/overview.map&layer=denmark&mode=tile&tilemode=gmap&tile=$2+$3+$1 [PT,L]

The PT (passthrough) flag is set to enable us to refer to /cgi-bin/ which is not located under DocumentRoot, but is ScriptAlias'ed in place.

If one want to move the rules to a virtual host configuration instead of using a .htaccess files the different path in the different RewriteRule expressions must be prepended with en slash, e.g.:

RewriteRule ^/fugro2005...

mod_cache

Apache's mod_cache and mod_disk_cache (part of Apache Core in version 2.2, not enabled per default) is configured to cache requests to mapserv. From the virtual host block:

CacheEnable disk /cgi-bin/mapserv
CacheIgnoreNoLastMod On
CacheMaxExpire 2592000

MaxExpire is set to 2592000 seconds (30 days). Whatever the value of a resource's Expire time in the HTTP header the cache does not make use of older files. Default is only one day so this is a relevant value to raise.

The cache is maintained and vacuumed by htcacheclean. In Debian this is configured in /etc/default/apache2. Here it is possible to adjust the cache max size, e.g.:

HTCACHECLEAN_SIZE=5000M

mod_expires

mod_cache is only caching files with query string (where "?" is part of the internal URL, including all tile requests), if an Expire header is present. Mapserver does not output this header, meaning that we have to add it ourselves.

mod_expires (not enabled per default) gives os this opportunity, based on file type.

In the virtual host block we add the following directives:

ExpiresActive On
ExpiresByType image/jpeg A2592000

Here we ask Apache to add an Expires header on image/jpeg output (including the files generated by mapserver - given that we are generating JPEG files), set to 2592000 seconds (= 30 days) after the time of the client request.

This way we ensure that the tiles have an Expires header needed for mod_cache to cache the tile.