CGImap

From OpenStreetMap Wiki
Jump to navigation Jump to search
openstreetmap-cgimap
CGImap screenshot.png
Author: See list of contributors
License: GNU General Public License v2 (free of charge)
Platform: Ubuntu/Debian
Version: 2.0.0 (2024-08-24)
Language: English
Source code: https://github.com/zerebubuth/openstreetmap-cgimap
Programming language: C++17

CGImap is a re-implementation of selected performance critical parts of the OSM API 0.6 in C++ (in productive use)

CGImap is a C++ implementation of selected performance critical parts of the OSM API v0.6 using the FastCGI interface. It was originally developed to overcome performance limitations present in the openstreetmap-website Rails application, including but not limited to large memory consumption, slow XML parsing, overall slow execution.

As CGImap doesn't manage any database schemas and tables on its own, it can only be deployed alongside the openstreetmap-website application to replace selected API endpoints by their performance optimized counterparts, see Section Production Deployment. CGImap directly interacts with the APIDB, and completely bypasses the openstreetmap-website Rails application.

An important design principle is that for every feature, a Rails port counterpart needs to be implemented first, which doubles as a reference implementation.

Features

Currently, CGImap implements 25 OSM API v0.6 endpoints. This roughly corresponds to 40% of the endpoints implemented on Rails. In production, only 19 endpoints are served by CGImap (Chef repo link to the Apache configuration), all other endpoints are handled by Rails. In particular, single element create/change/delete operations, as well as GPS traces, user data and the map notes API are currently out of scope for CGImap.

  • the "map" API call
  • single node, way and relation fetches
  • multiple node, way and relation fetches
  • the "full" way and relation calls
  • changeset metadata downloads, including discussions
  • single node, way and relation history calls
  • single node, way and relation specific version fetches
  • multiple node, way and relation specific version fetches
  • changeset downloads
  • changeset uploads

(Always check the project GitHub repo for an up-to-date list.)

Supported output formats include OSM XML and OSM JSON. Since version 2.0.0 only OAuth 2.0 can be used for user authentication.

Deviations from Rails openstreetmap-website

  • Diff upload rejects empty keys.
  • Diff upload relies on SQL to create new object version and changeset timestamps, thereby ignores the time reported by the frontend server.
  • Feature Node, way and relation multi-selection calls is not yet available on Rails (Pull request)
  • There is currently no limit for osmChange uploads on the Rails port. CGImap has a configurable default limit of 50MB, based on the uncompressed message.
  • CGImap supports IP-based and user id based rate limiting, with different limits for "normal" users, and moderators. Rails does not have a comparable rate limiting in place.
  • CGImap send an RFC 9110 compliant HTTP Retry-After header, in case too much data has been downloaded. Rails does not send this HTTP header.

Shared code with openstreetmap-website

Since release 0.8.10, CGImap reuses the database function api_rate_limit, which has been introduced in "Add the ability to rate limit edits", and is managed in the Rails repo. Changeset upload rate limiting is off by default, and needs to be enabled by setting the CGIMAP_RATELIMIT_UPLOAD environment variable to true (or the respective --ratelimit-upload command line parameter).

Since release 0.9.3, another database function api_size_limit is used to restrict the size of a changeset bounding box. This feature has been introduced in "Add the ability to limit changeset size".

Installation

Installation instructions are available on the project's GitHub repo.

Trivia

Customer quotes

  • We have some hidden magic that makes the website faster for editing. [...] A few years ago we implemented that and you could notice how much quicker the API was.[1]
  • I got it to run as an editable map yesterday but immediately discovered I need cgimap to make it anywhere close to usable.[2]
  • The cgi-map implementation of the diff upload endpoint is blazingly fast[3]
  • CGImap has dramatically improved my experience as a contributor to OpenHistoricalMap. Before, I could barely upload boundary relations without JOSM timing out and retrying, and using iD sometimes felt like reverting to a dialup connection. Now I can upload and download without going out for a walk while I wait.[4]

Faster implementation → less hardware needed:

  • From today the http://OpenStreetMap.org mapping API is now >30% more energy efficient with no reduction in performance. We have removed the need for separate backend API servers.[5]

All time record for hourly changeset uploads

A few days before the introduction of the rate limit for uploading changesets, we set a new record for hourly changeset uploads. With a compressed size of 179 MB, it was even larger than many daily diffs. Needless to say, these changes were mainly reverting vandalism and not regular mapping activities. With rate limits in place now, even a moderator / import account cannot upload more than 1 million changes per hour.

Prometheus reported an average response time of <1s for most of the hour. While there was a bit of an increased load on the database server (link), this upload event wasn't noticeable on the three frontend servers.

Notes and references

  1. ↑ https://2022.stateofthemap.org/sessions/D8XYDN/
  2. ↑ https://en.osm.town/@mvexel/110373492026784808
  3. ↑ https://twitter.com/sp8962/status/1120004104061894656
  4. ↑ https://github.com/OpenHistoricalMap/issues/issues/588
  5. ↑ https://twitter.com/OSM_Tech/status/1304032683161518090
Retrieved from "https://wiki.openstreetmap.org/w/index.php?title=CGImap&oldid=2747229"