FR:Osm2pgsql

From OpenStreetMap Wiki
Jump to navigation Jump to search
Osm2pgsql - Other languages

osm2pgsql est un programme en ligne de commande convertissant les données OpenStreetMap en bases de données PostgreSQL postGIS-activé.

De telles bases de données sont souvent utilisées pour obtenir un rendu visuel des données via Mapnik, comme PostgreSQL est un des formats les plus efficient et flexible Mapnik peut ainsi utiliser en termes de requête de grands volumes de données. Pour davantage d'information sur l'utilisation d'osm2pgsql pour le rendu des données OpenStreetMap avec Mapnik, consultez la page Mapnik.

Sa page d'accueil est : https://github.com/openstreetmap/osm2pgsql Elle n'est actuellement pas maintenue par une seule personne mais par une communauté de contributeurs et Osm2pgsql ne maintient pas de versions 'stable' et en 'développement' maos les réalisations les plus récentes devraient être pouvoir utilisée en environnement de production. Jusqu'à Avril 2013, il était hébergé sur svn.

Il est hautement recommandé d'utiliser une version publiée depuis le 2 Septembre 2, 2012, qui a un support pour 64-bit_Identifiers.

En accord avec https://github.com/openstreetmap/osm2pgsql/blob/master/CONTRIBUTING.md#documentation "La documentation utilisateur est stocké à docs/. Les pages du wiki d'OpenStreetMap sont connues pour être non fiables et dépassées."

OS Support

osm2pgsql est disponible pour Linux, Mac OS X, et Windows. La plupart des utilisateurs osm2pgsql utilisent Ubuntu or une distribution basé sur Debian et vous trouverez une documentation plus complète et plus à jour et un support sur celle_ci.

Installation

A partir d'une source

Les conditions d'utilisation sont listées sur the osm2pgsql README. Les principales dépendances à porter attention pour d'anciennes distributions sont un compilateur C++11 et Boost 1.50 ou plus récent.

A partir du package manager

osm2pgsql est packagé sur la plupart des systèmes d'exploitation Linux. Si vous utilisez un ancienne distribution, vérifier que vou avez au moins la version 0.82.0 avec osm2pgsql -h

Pour Debian ou Ubuntu

apt-get install osm2pgsql

Fedora

dnf install osm2pgsql

openSUSE

Pour openSUSE 11.3 et plus récent : Tout d'abord ajoutez le Geo package repository (adaptez l'URL à la correcte version openSUSE): 

sudo zypper ar http://download.opensuse.org/repositories/Application:/Geo/openSUSE_11.3/ "Geo"
sudo zypper refresh

Ensuite installez osm2pgsql: 

sudo zypper install osm2pgsql

Arch Linux

Construisez le osm2pgsql-git package à partir de AUR. (Téléchargez le tarball et compilez/installez avec 'makepkg', ou utilisez un assistant AUR tel que 'yaourt'.)

FreeBSD

pkg install converters/osm2pgsql

Windows

Binary

Warning
 Cette méthode ne vous fournira pas une installation fonctionnelle pour les raisons ci-dessous. Use Cygwin instead

A partir de la Source

Pour de natives compilation Windows avec uniquement de petite sources d'ajustements il est recommandé d'utiliser un compilateur compatible C99 comme le compilateur Intel.

Via MinGW, il y a actuellement beaucoup de problèmes à résoudre lors de la construction d'osm2pgsql:

Le script configuré osm2pgsql ne trouve pas zlib et autres bibliothèques

Le script configuré créé par autogen ne fonctionne pas correctement parce que les paramètres pour la définition de bibliothèques devant à relier (e.g. -lz for libz) ne se trouvent pas au bon emplacement. Ils doivent être positionnés à la fin de la ligne de commande. En ajoutant y manuellement les paramètres correspondant.

Solution: Editer le script de configuration et ajouter les bibliothèques à relier manuellement : 

ac_link='$CXX -o conftest$ac_exeext $CXXFLAGS $CPPFLAGS $LDFLAGS conftest.$ac_ext $LIBS -lz >&5'

Note: Ajoutez le paramètre de débogage d à autogen.sh (autoreconf -vfid) est extrêmement utile dans cette situation pour trouver quelle ligne éditer.

Problèmes de liaison 

build_geometry.cpp:206: undefined reference to `geos::geom::Coordinate::Coordinate(double, double, double)'

Workaround: Edit build_geometry.cpp and add #define GEOS_INLINE

Used versions: libxml2-2.7.8, zlib-1.2.5, proj-4.7.0, geos-3.2.2

Jburgess, who compiled the newest windows binary (2010-04-10), used mingw32 to cross-compile the windows binary from a Linux host.

Cygwin

Cross-compilation problems with osm2pgsql are documented on github. Whilst these problems are being worked on, an alternative compilation path for windows users targeting Cygwin has been used.

This can be run directly in Windows as all relevant DLLs are included in the distribution. Alternatively it can be used under Cygwin within Windows.

Binary

Download the zipped osm2pgsql Cygwin package (latest version (95) here: https://vanguard.houghtonassociates.com/browse/OSM-OSM2PSQL-95/artifact).

Unzip this into a directory usually cygwin_package, but perhaps better called osm2pgsql.

You can run osm2pgsql within this directory and/or add the directory to your PATH variable. (In practice I have a single windows directory for core OSM apps (osmosis, osmconvert, osmfilter, osm2pgsql, ogr2osm etc.) and copy apps and their libraries there.

(A quick hack if running this under Cygwin itself which I've used is simply to copy the .exe and .dll files to /usr/bin, some of the .dlls will already exist. Osm2pgsql will then be in the path for Cygwin.)

See also notes under the Windows Binary section regarding PATH and other environment variables.

Mac OS X

With Homebrew

brew install osm2pgsql

Osm2pgsql schema

Information about the schema created by Osm2pgsql can be found here: Osm2pgsql/schema.

Import style

Aspects of how osm2pgsql converts OSM data into PostgreSQL tables can be configured via a style file. The default style file that is installed with osm2pgsql is suitable for rendering the standard OSM Mapnik style or similar styles. It also contains the documentation of the style syntax. With a custom style file, you can control how different object types and tags map to different columns and data types in the database.

The style file is a plain text file containing 4 columns separated by spaces. As each OSM object is processed it is checked against conditions specified in the first two columns. If they match, processing options from the third and fourth columns are applied.

  1. OSM object type: can be node, way or both separated by a comma. way will also apply to relations with type=multipolygon, type=boundary, or type=route; all other relations are ignored by osm2pgsql.
  2. Tag: the tag to match on. If the fourth column is linear or polygon, a column for this tag will be created in each of the point, line, polygon, and road tables.
  3. PostgreSQL data type: This specifies how data will be stored in the tag's postgresql table column. Possible values are text, int4, real. If the fourth column is delete or phstore, this column has no meaning and should just be set to text.
  4. Flag: Zero or more flags separated by commas. Possible values:
    • linear: Specifies that ways with this tag should be imported as lines by default, even if they are closed. Other conditions can still override this. See the Osm2pgsql/schema#planet_osm_line schema documentation for details on how route relations are handled.
    • polygon: Specifies that closed ways with this tag should be imported as polygons by default. This will override any linear flags that would apply to the same object. Closed ways with area=yes and closed relations with type=multipolygon or type=boundary will be imported as polygons even if no polygon flag is set. Non-closed ways and closed ways with area=no will always be imported as lines.
    • nocolumn: The two flags above automatically create a column for the specified tag. This flag overrides this behaviour such that no column is created. This is especially useful for hstore, where all key value data is stored in the hstore column such that no dedicated columns are needed.
    • phstore: The same as polygon,nocolumn for backward compatibility
    • delete: Prevents the specified tag (but not the entire object) from being stored in the database. Useful for tags that tend to have long values but will not be used for rendering, such as source=*. This flag only affects --slim mode imports.
    • phstore: Behaves like the polygon flag, but is used in hstore mode when you do not want to turn the tag into a separate PostgreSQL column.
    • nocache: This flag is depreciated and does nothing.

The style file may also contain comments. Any text between a # and the end of the line will be ignored.

Special 'tags'

There are several special values that can be used in the tag column (second column) of the style file for creating additional fields in the database which contain things other than tag values.

  • way_area: Creates a database column that stores the area (calculated in the units of the projection, normally Mercator meters) for any objects imported as polygons. Use with real as the data type.
  • z_order: Adds a column that is used for ordering objects in the render. It mostly applies to objects with highway=* or railway=*. Use with int4 as the data type.
  • osm_user: Adds a column that stores the username of the last user to edit an object in the database.
  • osm_uid: Adds a column that stores the user ID number of the last user to edit an object in the database.
  • osm_version: Adds a column that stores the version of an object in the database (ie, how many times the object has been modified).
  • osm_timestamp: Adds a column that stores the date and time that the most recent version of an object was added to OpenStreetMap.

To use the osm_user, osm_uid, osm_version, and osm_timestamp tags, you must use the osm2pgsql option --extra-attributes when importing.

Avoiding pitfalls

osm2pgsql is a specialized tool and there's some behavior hardcoded into it that you should know about.

Coastlines

The natural=coastline tag is suppressed by default, even if you import the natural=* key. The main mapnik map renders coastlines from shapefiles so it does not need them. You can use the --keep-coastlines parameter to change this behavior if you want coastlines in your database.

Usage

For basic usage, see the man page of osm2pgsql (man osm2pgql) and the README found on its github page.


Before running osm2pgsql, you first must create your postgresql database and enable postgis on it.


Basic usage:

osm2pgsql -s -U postgres -d nameofdatabase /file/path/toosm/fileorpbf/name.osm

In the example above, postgres is the user of the database, the user enabled slim mode (generally recommended).

The above sample is feasible for someone looking to export a city's OSM data into a postGIS database. For more advanced and larger datasets, read the Optimization section.

Additional parameters:

-G|--multi-geometry Generate multi-geometry features in postgresql tables.

--flat-nodes flat-nodes is an alternative node store in a file which uses ~20GB (Oct 2013) instead of 100GB in Postgresql. It is also faster during import and during diff updates, and thus can only be used in slim mode. Put this file on a SSD, if you can.

Optimization

Best practices

Optimising the Mapnik/osm2pgsql Rendering Toolchain 2.0 @ SOTM 2012

Slim mode

osm2pgsql has two main modes of running - normal and slim mode.

It is highly recommended to run osm2pgsql in slim mode. Some important features (including incremental updates (planet diffs, the initial load to populate the track tables, and proper evaluation of multipolygons) only work in slim mode.

The normal mode uses RAM for intermediate storage, Slim mode uses object tracking tables

  • planet_osm_nodes
  • planet_osm_ways
  • planet_osm_rels

in the database on-disk. You must use slim-mode for planet imports on 32-bit systems, since there are too many nodes to store in RAM otherwise. This limitation doesn't apply to 64-bit systems.

One benefit of not using slim mode is that osm2pgsql is a bit faster since it doesn't read from the DB until the index creation.

Implement slim mode by using the '-s' option: "osm2pgsql -s -d ...."

Parameters

  • Large imports into PostGIS are very sensitive to maintenance and monitoring configuration: it is smart to increase the value of checkpoint_segments so that autovacuum tasks don't slow down imports. This can be done by editing postgresql.conf.

See the Postgres Wiki for reference.

  • osm2pgsql relies much on its node cache during import. If the nodes do not fit into the cache in slim mode it needs to do database lookups which slow down the process. (Without slim mode, it fails if the nodes do not fit in the cache). Use enough cache so all nodes are cached. -C 22000 seems to do the job, even if that means you have to configure more swap space.
    • Without slim mode, there is also a "way" cache which takes up about as much space as the node cache does, but is not charged against -C. In slim mode, there is no "way" cache.

Fast disks

The bottleneck is usually the I/O when running osm2pgsql with --slim or even without it.

Benchmarks

Please see /benchmarks for general benchmarking information.

Updating Data in Database

osm2pgsql is one of the tools used to keep OSM data updated with changes to the OpenStreetMap servers in your own PostgreSQL database. Please see Minutely_Mapnik for more information.

Bug reports

Please report bugs to the github tracker.

hstore

Hstore is for sets of key/value pairs. As associative array datatype, just like a hash in perl or dictionary in python.

This should come in handy especially for rarely used tags. Using hstore, one can use any tag in sql queries like this: 

gis=> select count(*) FROM planet_osm_point where ((tags->'man_made') = 'tower');
 count
-------
  447
(1 Zeile)

-k|--hstore		Generate an additional hstore (key/value) column to  postgresql tables

install into postgresql

Postgresql 9.1 and later

This sql code will install the extension in your current database[1] : 

CREATE EXTENSION hstore;

Voir également

  • Manually building a tile server - Also explains how to create the database for osm2pgsql, import data, etc
  • PostGIS/Installation - Explains how to create the database for osm2pgsql
  • Mapnik - Explains how to import OSM data
  • High Road - collection of Postgres views that make rendering roads from Osm2pgsql schema easier and better-looking

Alternatives to Osm2pgsql


  1. ↑ [1]
Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=FR:Osm2pgsql&oldid=1545666"