Talk:Osmosis
Locale depending decimals
On my xp box with Slovenian locale osmosis made an extract from planet's node with decimal points:
<node id="6968827" timestamp="2007-11-06T20:56:47Z" lat="46.0544422" lon="14.5046166" />
into a node with decimal commas:
<node id="6968827" timestamp="2007-11-06T20:56:47Z" lat="46,0544422" lon="14,5046166" />
Trying to import such extract into mysql (with osmosis) it gave me:
Exception in thread "Thread-1-read-xml" java.lang.NumberFormatException: For input string: "46,0544422"
at sun.misc.FloatingDecimal.readJavaFormatString(Unknown Source)
at java.lang.Double.parseDouble(Unknown Source)
at com.bretth.osmosis.core.xml.v0_5.impl.NodeElementProcessor.begin(NodeElementProcessor.java:62)
at com.bretth.osmosis.core.xml.v0_5.impl.OsmHandler.startElement(OsmHandler.java:91)
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source)
at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)
...
IMO OSM xml should always (regardless of current locale) be formatted with decimal dots (us locale), so that they are portable.
Temporary workaround is to include commandline parameters
-Duser.country=US -Duser.language=en
but permanent one something in the line of
Locale.setDefault(new Locale("en","US"));
during osmosis initialization.
- --Stefanb 07:21, 18 November 2007 (UTC)
- Fixed by User:Brett in version 0.24. Tnx! --Stefanb 06:57, 19 November 2007 (UTC)
Losless version of enableDateParsing
enableDateParsing option looses timestamps, however it could treat timestamps as strings, copy them to output as string values without really parsing them into a java Date object. This should still be quite fast AND lossless. --Stefanb 21:09, 4 December 2007 (UTC)
- Of course this would only make sense if no comparing of timestamps is needed. Or it could be implemented as lazy date parsing (handling them as strings until comparing is needed) --Stefanb 13:13, 15 January 2008 (UTC)
- Yes, I was thinking lazy parsing would be the way to go, and if they're unused in the pipeline, just write out the original timestamp string. Probably would have to change the constructor to accept a string instead of a date, and add another function to get the raw date string. --SiliconFiend 04:36, 16 January 2008 (UTC)
- Only now i noticed changes.txt in SVN changesets 8341 and 8342 mentioning:
- Enhanced the Entity class to allow dates to be represented as strings to avoid parsing overhead. Updated xml tasks to utilise this feature.
- but i fail to see any speed boosts (compared to my build of version 0.24.3) or documentation changes about it. How does one use that? Documentation for enableDateParse still basically says "no timestamps or poor performance" --Stefanb 00:36, 22 November 2008 (UTC)
Producing empty timestamps
If the elements of input file don't have the optional(!) timestamp attribute (eg from Srtm2Osm) the resulting elements will have a non-sense (and probably invalid) timestamp="" attribute on the output. Not a major problem, but worth knowing and fixing sometime. --Stefanb 10:49, 17 January 2008 (UTC)
gpsbabel produces 0.5 files without a timestamp too. As of osmosis-0.31.2 this causes a crash.
osmosis-0.31.2/bin/osmosis --read-xml-0.5 Srtm2Osm/srtm3-full-route.osm --migrate --wx Srtm2Osm/srtm3-full-route06.osm
org.openstreetmap.osmosis.core.OsmosisRuntimeException: The entity timestamp attribute is missing. at org.openstreetmap.osmosis.core.domain.common.UnparsedTimestampContainer.<init>(UnparsedTimestampContainer.java:34) at org.openstreetmap.osmosis.core.xml.common.BaseElementProcessor.createTimestampContainer(BaseElementProcessor.java:90) at org.openstreetmap.osmosis.core.xml.v0_5.impl.NodeElementProcessor.begin(NodeElementProcessor.java:65) at org.openstreetmap.osmosis.core.xml.v0_5.impl.OsmHandler.startElement(OsmHandler.java:94)
Others have seen this too http://trac.openstreetmap.org/ticket/2043
Filter out relation
Hi, I can't find an option to filter out all relations from the output. --chris66 15:39, 13 November 2008 (UTC)
Bounding Box
when cutting out a rectangular area (--bounding-box) is it possible to write this bounding box to the output file in the header? --chris66 15:45, 13 November 2008 (UTC)
Also is there any chance that Bounding Box could deal with change sets rather than just raw xml? --Sjk 01:21, 19 November 2008 (UTC)--sjk
Osmosis running forever with completeWays=yes?
I am trying to cut 10 areas out of the planetfile for germany. When I set the parameter completeWays=yes for these areas, osmosis (V0.29) seems to run forever. After 8,5 hours of CPU time, osmosis had created a number of large temporary files, one output file was at about 15% the expected size and all others were still empty. Without the parameter completeWays=yes the job is completed in less than 30 minutes.
Is completeWays=yes broken? Or does it really take this long to process? Is there anything I can do? I want the cut areas to be seamless, that is the reason I switched on completeWays=yes. --Nop 19:09, 1 February 2009 (UTC)
where find specification ?
artice say osmosis deal with database and file !! is there some more details in some documentation ? : all jdbc drivers not only mysql ? User:France-59-valenciennes 15:51 , 8 February 2009 (UTC)
Import into PostGreSQL fails
I'm trying to use osmosis to import direct into postgresql.
I installed Osmosis using:
wget http://gweb.bretth.com/osmosis-latest.tar.gz tar xvfz osmosis-latest.tar.gz cd osmosis-0.30 bin/osmosis
The notes say cd to osmosis-0.29. Downloaded version is osmosis-0.30 - I guess not a problem.
Postinstallation, when I run this command:
- $ bin/osmosis --read-xml file="/root/south-090218.osm" --write-pgsql host="localhost" database="osmtest1" user="postgres" password="..."
I get this error:
24-Feb-09 2:56:32 com.bretth.osmosis.core.Osmosis run
INFO: Osmosis Version 0.30
24-Feb-09 2:56:33 com.bretth.osmosis.core.Osmosis run
INFO: Preparing pipeline.
24-Feb-09 2:56:33 com.bretth.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
24-Feb-09 2:56:33 com.bretth.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
24-Feb-09 2:56:33 com.bretth.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-read-xml failed
java.lang.NoClassDefFoundError: org.postgresql.jdbc4.AbstractJdbc4Connection
at java.lang.Class.initializeClass(libgcj.so.81)
at java.lang.Class.initializeClass(libgcj.so.81)
at org.postgresql.Driver.makeConnection(Driver.java:386)
at org.postgresql.Driver.connect(Driver.java:260)
at java.sql.DriverManager.getConnection(libgcj.so.81)
at java.sql.DriverManager.getConnection(libgcj.so.81)
at com.bretth.osmosis.core.pgsql.common.DatabaseContext.getConnection(DatabaseContext.java:85)
at com.bretth.osmosis.core.pgsql.common.DatabaseContext.createStatement(DatabaseContext.java:184)
at com.bretth.osmosis.core.pgsql.common.SchemaVersionValidator.validateDBVersion(SchemaVersionValidator.java:72)
at com.bretth.osmosis.core.pgsql.common.SchemaVersionValidator.validateVersion(SchemaVersionValidator.java:52)
at com.bretth.osmosis.core.pgsql.v0_5.PostgreSqlWriter.initialize(PostgreSqlWriter.java:284)
at com.bretth.osmosis.core.pgsql.v0_5.PostgreSqlWriter.process(PostgreSqlWriter.java:1012)
at com.bretth.osmosis.core.xml.v0_5.impl.NodeElementProcessor.end(NodeElementProcessor.java:103)
at com.bretth.osmosis.core.xml.v0_5.impl.OsmHandler.endElement(OsmHandler.java:109)
at gnu.xml.stream.SAXParser.parse(libgcj.so.81)
at javax.xml.parsers.SAXParser.parse(libgcj.so.81)
at javax.xml.parsers.SAXParser.parse(libgcj.so.81)
at com.bretth.osmosis.core.xml.v0_5.XmlReader.run(XmlReader.java:109)
at java.lang.Thread.run(libgcj.so.81)
Caused by: java.lang.ClassNotFoundException: java.sql.SQLClientInfoException not found in gnu.gcj.runtime.SystemClassLoader{urls=[file:/root/osmosis-0.30/osmosis.jar,file:/root/osmosis-0.30/lib/mysql-connector-java-5.0.7-bin.jar,file:/root/osmosis-0.30/lib/postgresql-8.3-603.jdbc4.jar,file:/root/osmosis-0.30/lib/postgis_1.3.2.jar], parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}
at java.net.URLClassLoader.findClass(libgcj.so.81)
at gnu.gcj.runtime.SystemClassLoader.findClass(libgcj.so.81)
at java.lang.ClassLoader.loadClass(libgcj.so.81)
at java.lang.ClassLoader.loadClass(libgcj.so.81)
at java.lang.Class.forName(libgcj.so.81)
at java.lang.Class.initializeClass(libgcj.so.81)
...18 more
24-Feb-09 2:56:34 com.bretth.osmosis.core.Osmosis main
SEVERE: Execution aborted.
com.bretth.osmosis.core.OsmosisRuntimeException: One or more tasks failed.
at com.bretth.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)
at com.bretth.osmosis.core.Osmosis.run(Osmosis.java:85)
at com.bretth.osmosis.core.Osmosis.main(Osmosis.java:30)
Advice appreciated.
User:Nerdling 15:06, 24 February 2009
- There is an osmosis-dev mailing-list for this kind of questions. In your case you did not add the postgres-jdbc-driver to your classpath. --MarcusWolschon 07:58, 25 February 2009 (UTC)
Maybe this is just user error (perhaps i need to escape it?), but passing a keyValueList that contains a key with a namespace kills osmosis without a whimper. e.g. osmosis --read-xml file="cambridge.osm" --node-key-value keyValueList="gnis:reviewed.no,building.yes" --write-xml file="filtered.osm"
User:Pouletic 02:09, 4 April 2009
Procedure to Import Data into PostgreSQL
What Java distributions will work?
I'm not really familar with java. Using Debian Lenny, I have a confusing choice of different java-distros. At least, there is:
- OpenJDK
- GCJ
- Sun JDK 6.0 (and even 5.0)
In Osmosis PostGIS Setup, there is an example using openjdk-6-jdk. Following Osmosis#Notes, the favorite distro seems to be Sun JDK 6.0. Could anybody give me a hint what would be the best to install?
Concerning PostGIS: I have experienced many oddities in PostGIS. So, is it worth to use the PostGIS storing scheme or is regular PostreSQL without PostGIS sufficient?
Thanks for any help. -- Hansm 09:57, 4 July 2009 (UTC)
- It's possible all of those will work. I don't think osmosis is very demanding in terms of java features, compared to many GUI apps for example... but then if you want (or may in the future want) other java apps running trouble free, I suspect a Sun JDK will be the best recommendation. More general java info on the Java Runtime Environment page -- Harry Wood 09:44, 8 November 2011 (UTC)
bandwidth exceeded
http://gweb.bretth.com seems to be having some bandwidth trouble.
"The bandwidth or page view limit for this site has been exceeded and the page cannot be viewed at this time. Once the site is below the limit, it will once again begin serving as normal."
-- Harry Wood 19:59, 21 July 2009 (UTC)
- There's a download mirror here (maybe only temporary) : http://download.tuxfamily.org/flossk/trunk/dist/ -- Harry Wood 10:52, 29 July 2009 (UTC)
Not working on windows. Batch recursion error
Trying to use it on Windows Vista from the command line running osmosis.bat I get:
C:\server\installers\osmosis-0.31.2\bin>osmosis ****** B A T C H R E C U R S I O N exceeds STACK limits ****** Recursion Count=650, Stack Usage=90 percent ****** B A T C H PROCESSING IS A B O R T E D ******
With 0.31.2 and I think 0.31.0/1 (not sure which - there are no version numbers in there, but the .bat file is different).
Any suggestions on how to actually get it to work? (I've not changed anything) Thanks. Asdf 15:34, 3 August 2009 (UTC)
- Hehe. Awesome error message. Not sure what's causing that exactly, but there's been a few problems with the windows bat script. Working from the latest Osmosis in subversion is probably a good idea, or just take a look at the current version of the bat script.
- See also this forum post where I've pasted in a full java command which should work (bypassing the bat script).
- Note that even when you get it working, you can't process recent bzipped planet files very easily with windows as far as I can tell. See Osmosis#Notes
- -- Harry Wood 17:07, 3 August 2009 (UTC)
- I get the same error message (amongst many when trying to use osmosis on Windows 7 x64, has the above been solved? I tried with latest but would run into that error nevertheless. Also no luck on ubuntu for me, seems I'm too stupid for osmosis. 6 month ago osmosis would run fine on my pc however. Don't have any old version anymore, 029 works in XP, but not in 7. On Ubuntu no luck with any version.--Extremecarver 16:00, 3 September 2009 (UTC)
- Same here - the same error message on my Vista --Breki 19:35, 16 September 2009 (UTC)
- I get the same error message (amongst many when trying to use osmosis on Windows 7 x64, has the above been solved? I tried with latest but would run into that error nevertheless. Also no luck on ubuntu for me, seems I'm too stupid for osmosis. 6 month ago osmosis would run fine on my pc however. Don't have any old version anymore, 029 works in XP, but not in 7. On Ubuntu no luck with any version.--Extremecarver 16:00, 3 September 2009 (UTC)
Cleanup Request
Hi, if it's ok I would like to clean up the osmosis related pages a bit, because for me it was a mess to find some more advanced things. I suggest to:
- move Benchmarking to a sep /Benchmarking
- move all examples to /Examples
- splitting Detailed usage into Storage/Manipulation/Pipeline features and add examples there?
- add a sketch showing pipe concept
--!i! 
11:17, 15 February 2011 (UTC)
If anybody knows if Using_planet.osm is still up to date feel free to label it as outofdate or remove it --!i! 21:01, 23 February 2011 (UTC)
- Please don't alter it as it stands. In particular do not remove examples and do not split Detailed usage into subpages. The pages are large and contain a lot of information: they are frequently used as core manual pages. As such they are probably more useful than many wikipages. SK53 19:10, 7 April 2011 (BST)
- I agree with SK53. There's different ways it could be organised but the Osmosis/Detailed Usage page works reasonably well as is. It's also quite carefully maintained by User:Brett (The main osmosis developer) as the primary documentation, and we don't want to get in his way doing that. No need to fiddle with it.
- The examples on the main Osmosis page, I'm thinking perhaps some of them should be moved onto Osmosis/Examples, but not all. It makes sense to have some some simple examples there for quick prominent access. In general the Osmosis/Examples page could be fleshed out with more useful content. It's currently has a couple of quirky how-tos.
- Sketch showing pipe concept is good idea
- Overall though I feel that this page isn't in need of a cleanup all that much. Right now the most messy thing about this page is the cleanup label! So I'm going to remove it.
- -- Harry Wood 15:56, 29 June 2011 (BST)
Windows piping
In the bounding box extract example, it is suggested to use CONIN$ instead of /dev/stdin on a windows command prompt. This does not work for me (Win7) and I could not find a good reference to using this variable. Is there any way to do this using Win7? -- Martijn van Exel 14:30, 28 February 2011 (UTC)
- fixed the example. use a dash for stdin. You might want to google for pipebuf.exe to speed up the pipe even more. --Stephankn 19:47, 6 March 2011 (UTC)
Importing "Something" into clean PostgreSQL/PostGIS database
Hello,
i have problems to follow the examples. While i am trying to start: 'osmosis --read-xml file="myosm.osm" --write-apidb host="localhost" database="postgis" user="postgres" password="xxx"' i get following errors.
The database is fresh and clean. My OS is Win x64
Here is the exception trace:
SCHWERWIEGEND: Thread for task 1-read-xml failed org.openstreetmap.osmosis.core.OsmosisRuntimeException: Unable to create resultset. at org.openstreetmap.osmosis.apidb.common.DatabaseContext.executeQuery(DatabaseContext.java:429) ...etc... at java.lang.Thread.run(Unknown Source) Caused by: org.postgresql.util.PSQLException: FEHLER: Relation +schema_migrations½ existiert nicht Position: 21 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062) ...etc... at org.openstreetmap.osmosis.apidb.common.DatabaseContext.executeQuery(DatabaseContext.java:424)
User:Asdfasdfasdf - 13:47, 1 May 2011
- You pasted in a massive stacktrace. I've thinned it down to just the relevant error messages. I'm pretty sure the key error message is "FEHLER: Relation +schema_migrations½ existiert nicht"
- I dont actually know the answer to your question, but I guess you need some tables in place prior to running osmosis. I've not tried doing this myself. I'm sure somebody else can help you, but you might have more luck using a different Contact channel.
- -- Harry Wood 11:25, 3 May 2011 (BST)
Wiki Osmosis organization
There's a lot of great Osmosis-related content on the wiki, but it seems like much of it is spread across several pages and is overlapping on many of those pages.
For example, the following pages cover some similar topics:
- Osmosis - includes installation, some use examples, tuning information, and leads with a pretty technical description of what Osmosis does
- Osmosis/Installation
- Osmosis/Examples - from the page: "This page gives some more obscure or convoluted examples of Osmosis usage"
- Detailed_Usage_0.41 - this seems more like "man osmosis" than "detailed use," as it primarily discusses command line options and not use cases
- Osmosis/Tuning -
- Using_planet.osm - includes optimization tips, osmosis use examples
I'd like to recommend a cleanup of the main Osmosis page, with the following structure:
- Introduction - less technical explanation - explain what it does and why it was built
- OSM data background - primary data types (xml, pbf), primary data stores used, sizes of files, etc.
- Osmosis basic tech background -
and make the following changes:
- Consolidate all installation information from Osmosis into Osmosis/Installation
- Combine the use examples from Osmosis and Using_planet.osm into a new Osmosis_Getting_Started page
- Move the tuning information from Using_planet.osm to Osmosis/Tuning
I realize these suggested changes might not seem small, but I believe they will streamline this content significantly, and consolidate similar information, which should hopefully reduce opportunity for conflicting information. It might also lighten the translation load if there's less redundancy.
Also - I'd be willing to do this work (in English : ) ).
Thoughts?
Jeffmeyer 07:57, 20 October 2012 (BST)
- What does "OSM data background" have to do with Osmosis? That would appear to duplicate content available elsewhere because it equally applies to all tools. --Tordanik 14:50, 20 October 2012 (BST)
- It has to do with the "why" of Osmosis - why does it exist in the first place? Why are there different storage formats? Why do we care about converting data formats? This shouldn't be a deep dive - that should certainly be linked elsewhere. I'm thinking of a sentence or two at most. As it is, although the page explains a little of what osmosis can do, it doesn't explain the why, at least imo. Any thoughts on the other suggestions? Jeffmeyer 21:16, 20 October 2012 (BST)
- Only minor feedback otherwise, e.g. Osmosis/Getting started would match the subpage structure used elsewhere. What I would like to see is something about the architecture and use as a library - something to point people with questions like these[1][2] -, but that might be a bit offtopic here, because it would be new content rather than a change of the pages' organization. Oh, and I think you should wait a while longer for feedback, or (maybe) even announce your intention on the osmosis-dev list to avoid that developers are surprised by your changes. Many users don't visit their wiki watchlist all the time. --Tordanik 23:12, 20 October 2012 (BST)
- Roger! Will do - ref: osmosis-dev list. I'm still getting the hang of how the OSM community process works. Also agreed on waiting a bit. Thanks for the help! Jeffmeyer 22:01, 21 October 2012 (BST)
How to configure temporary filespace?
I've just switched to using idTrackerType=Dynamic, and I'm noticing that Osmosis is throwing huge (700MB) files into /tmp (yes /tmp not $TMPDIR). Which is a problem when /tmp is only 1200MB and other things wish to use it. If Osmosis respected $TMPDIR like every other application, then it would be easy to point it somewhere with more space. So what's the magic to tell it where to put this huge file? --tms13 (talk) 16:13, 15 February 2013 (UTC)
Forgot to mention - the temporary files are not removed when Osmosis throws an exception. I expected a 'finally' to clear up this resource drain... --tms13 (talk) 12:39, 18 February 2013 (UTC)
- Not sure if this is helpful because it's not answering your question but... Which version of osmosis are you using? If you use the latest v0.42 you don't (I believe) need to use idTrackerType=Dynamic to solve the 64-bit Identifiers problems. -- Harry Wood (talk) 14:00, 18 February 2013 (UTC)
- Thanks, Harry. I'm currently using Osmosis Version 0.40.1 (Debian testing), and will remove the idTrackerType argument as soon as we get 0.42 (I only added it because osmosis starting failing on 64-bit ids last week). I don't think the file usage is specific to using idTrackerType=Dynamic, though; I'm half-certain that it does the same without. :( --tms13 (talk) 17:08, 18 February 2013 (UTC)
- And now here's the actual answer:
- export JAVACMD_OPTIONS="-Djava.io.tmpdir=/home/fred/tmp"
- Found this in a bug ticket here: https://trac.openstreetmap.org/ticket/3398
- -- Harry Wood (talk) 18:00, 18 February 2013 (UTC)
- Thanks again. I've now added
if [ -d "$TMPDIR" ]; then
JAVACMD_OPTIONS="$JAVACMD_OPTIONS${JAVACMD_OPTIONS+ }-Djava.io.tmpdir=$TMPDIR"
fi
Bugs
Where can bugs be logged (e.g. the problems reported at https://help.openstreetmap.org/questions/45185/osmosis-data-not-found ). There's and https://github.com/openstreetmap/osmosis also https://github.com/SomeoneElseOSM/osmosis - but there seems to be no issues list? --SomeoneElse (talk) 12:20, 11 September 2015 (UTC)
Password is compulsory?
I am using version 0.45 - for some reason, when I run osmosis --read-apidb database="osm" user="$USER" validateSchemaVersion="no" --write-pbf file="/path/to/planet_file.osm.pbf", it rejects the command insisting that my account must have a (PostgreSQL) password, even though I created my PostgreSQL user/role without any password. Why? --Ika-chan! (talk) 19:20, 21 January 2018 (UTC)