Skip to main content

Updating OSM data in PostgreSQL snapshot schema from latest pbf

Getting pbf files from Geofabrik and uploading them into your own datastore has been discussed in this blog earlier. In this blog, we are gonna see how to update the data, as OSM data changes every day by millions of controbutors worldwide. 

It is important to preseve the most recent pbf file in your hard drive, as we'll see in further.

The steps to be followed are the following,

  1. Download the latest pbf file as per your business requirement.
  2. Generate the changes between the last pbf and the earlier version.
  3. Apply the change to your PostgreSQL data store.

Let's follow this with a concrete example, 
  1. Assuming the last pbf we have applied is, india-20220122.osm.pbf dated Jan, 22, 2022.
  2. Download the most recent india osm pbf file, india-latest.osm.pbf
  3. Generate the change-log in xml format,
    4. osmosis-0.48.0/bin/osmosis --read-pbf file="india-latest.osm.pbf" --read-pbf file="india-20220122.osm.pbf" --derive-change --write-xml-change file=inddiff.osc
  4. Aplpy the changes in your local postgreSQL database,
    5. osmosis-0.48.0/bin/osmosis --read-xml-change file="inddiff.osc" --write-pgsql-change database=pgsnapshot host=localhost user=postgres password=postgres
With these steps completed, you should have updated data in your local database. 
     
Labels:

Comments

Post a Comment

Popular posts from this blog

Multimaster replication with Symmetric DS

Symmetric DS is an awesome tool for trigger based replication whcih works for all major database vendors, including but not limited to PostgreSQL, MySQL, MSSQL, Oracle and many others. Symmetric-DS is a java application and can execute on any platform on whcih JRE is available including Windows and Linux. Trigger based replication, in constrast to disk based (eg. DRBD ) or transaction log file shipping based or statement based , works by registering triggers on DMLs and sending the data thus generated to remote machines. Another very popular trigger based DB replication tool is Slony . Symmetric-DS in addition to being database agnostic also supports multi-master replication (MMR). MMR usecase involves multiple database nodes, connected in a pool with DML updates coming from any of them. This is different from the normal master-slave replication, where slaves are not expected to generate any data events, and the sole authority of database is the master. MMR requirement causes d...
1 comment
Read more

RabbitMQ and SSL

RabbitMQ is an AMQP provider i.e. it can reliably queue, service and maintain messages according to a range of policies and parameters. By default, it listens to plain old TCP connections and sends and receives messages over plaintext. This feature just works "out of the box". For users who wish to use SSL over TCP aka TLS, it requires a bit more work on their part. First, let's create a bunch of certificates and sign them with our own CA. For this, we'll use easyrsa3 . Easyrsa is a CLI tool to create, sign and manage your own certification authorities. It's maintained by OpenVPN team. Download easyrsa using your native package manager i.e. yum or apt-get $cp -Rp /usr/share/easy-rsa ~/easy-rsa-3   $cd ~/easy-rsa-3 $./easyrsa init-pki $./easyrsa build-ca $./easyrsa build-server-full broker [nopass] $./easyrsa build-client-full client1 [nopass] This creates three entities (collection of private keys, public keys and certificates) for a CA, a s...
Post a Comment
Read more

Motorola XT502 Custom ROMs

I purchased an Android Phone in the early days of it coming to Indian market and I able to afford it ( just got a job), during mid 2010. The recent and popular version was Éclairs. I just went up to the shop and bought a nice and shiny Motorola Quench XT3 (XT502). When in other places XT502 was having Andriod Donut, I got as a special offer Éclairs. I was happy as hell. The Droid A couple of year passes, and newer versions of Android came from Éclairs to Froyo to Gingerbread then to the bigger version upgrades like Honeycomb, Ice Cream Sandwich and then Jelly Bean. By the end of 2012, I was literally surviving on my Motorola with Éclairs. I had to upgrade, anyhow. Now more that Motorola denied any upgrades for Quench XT3 . A trivia, Android version names are taken from sugary desserts with lexicographic sequencing. So, I did up-gradation from Éclairs to Gingerbread using a custom ROM from Cyanogenmod . An excellent community of enthusiast who develop their own custom ROMs oft...
5 comments
Read more