User:Mmd/Overpass API/Performance Project 2016/Full Attic DB Setup

Summary

• Branch: https://github.com/mmd-osm/Overpass-API/tree/test754_threadupd
• Parallel DB update (see commit 1a575e)
• LZ4 compression

Prerequisites:

• Initial setup of database with first ODbL compliant planet is already done, dispatcher is up and running.

Multi Diff based update

• Update based on daily diffs
• Preparation: load daily diffs (instead of minutely diffs) from planet OSM server: nohup ./fetch_osc.sh 0 https://planet.openstreetmap.org/replication/day [osc_target_directory]
• Copy & adjust script apply_osc_to_db.sh to cater for up to 4 GB of .osc data: ( `du -m $TEMP_DIR | awk '{ print $1; }'` -le 4000 ) ]];.
• Run db update: nohup ./apply_osc_to_db.sh [osc_target_directory] 0 --meta=attic &
• One update_from_dir call will now process between 6-12 days of OSM data. Due to a peak memory consumption of > 15GB, at least 32GB main memory (!) is highly recommended.

• Total runtime for 1341 days OSM data: 4 days
• Parts of the db was moved to hard disk during the update (space constraints)
• Total db size (excl. areas): 173GB (after clone db!)

Comparison Single Diff based (left) vs. Multi diff based update (right)

Single Diff based update

Summary

• based on script fetch_osc_and_apply.sh
• update_database --flush-size=0 is essential to avoid database corruption -> adjust script accordingly!
• Frequent DB reorganization: update process was stopped several times to clone database and continue DB update on cloned DB -> massive reduction in DB size, up to 50 GB(!) during first run

• Total runtime for 1300+ days (starting September 2012 thru present) about 8-8,5 days.
• Total runtime without parallel processing / db reorganization: 13 days

• Initial full area creation: 4,5 hours (all on SSD) - (see comment)
  
• (v0.7.52 needs 13,5 hours for full area creation)

• Resulting DB size: ~180GB

Munin stats