Cynic's Blog

Getting pbf files from Geofabrik and uploading them into your own datastore has been discussed in this blog earlier. In this blog, we are gonna see how to update the data, as OSM data changes every day by millions of controbutors worldwide.  It is important to preseve the most recent pbf file in your hard drive, as we'll see in further. The steps to be followed are the following, Download the latest pbf file as per your business requirement. Generate the changes between the last pbf and the earlier version. Apply the change to your PostgreSQL data store. Let's follow this with a concrete example,  Assuming the last pbf we have applied is, india-20220122.osm.pbf dated Jan, 22, 2022. Download the most recent india osm pbf file, india-latest.osm.pbf Generate the change-log in xml format, osmosis-0.48.0/bin/osmosis --read-pbf file="india-latest.osm.pbf" --read-pbf file="india-20220122.osm.pbf" --derive-change --write-xml-change file=inddiff.osc Aplpy the changes...

Opensteetmap is the crowdsourced mapping of the planet and can be accessed at https://opensteetmap.org The map data is freely downloadable and can be hosted locally for any purpose whatsoever under GPL license. Map data is downloadable in multiple archive formats like pbf  (Protocolbuffer Binary Format) and others. OSM's internal data model is represented as shown in the ER diagram above, but when it's imported from the PBF, it has to be efficient, fast and easy to use. Primarily, OSM databases can either be, Snapshot or  Historical Snapshot databases don't contain historical infromation (for instance, how a river changed it's course of the years or how landuse of a forested land changed over the year) whereas, Historical database contains this information. Naturally, historical databases are much more complex and requires more effort. Various parameters on how to judge various data stores are, Updatability Transactibility Concurrent Read/Writability Snapshot/Historical...

Design Patterns listed out in GoF  Creational Pattern Abstract Factory Builder Factory Method Prototype Singleton Structural Pattern Adaptor Bridge Composite Decorator Facade Flyweight Proxy Behavioral Pattern Chain of Resposibility Command Interpreter Iterator Mediator Momento Observer State Strategy Template Method Visitor

For those who are familiar with Secugen Devices, might know that there are two function calls to initialize the device API.   The Init(long devName) and  The InitEx(long width, long height, long dpi) As per the official Javadoc, the first one is meant to " Initialize the SecuGen libraries using attached SecuGen fingerprint sensor. This method should be called after Open() is called."  while the second one is for " Initializes the SecuGen libraries using image parameters. Only 8bits per pixel raw images are supported."  For the same function, the C++ documentation says , "Use when running fingerprint algorithm module without a SecuGen reader." I had a use case wherein, I want to do the fingerprint verification/matching at server-side. This made me wonder if I can use the second initialization function at the server end without the device and use the fingerprint matching APIs. I tried using the aforementioned approach, but the InitEx() method was failing ...

PostgreSQL Server 1. Generate certificates Download easyrsa2 from github and extract it   # ./easyrsa build-ca # ./easyrsa build-server-full postgresql-server # ./easyrsa build-client-full postgresql-client   This will generate ca.crt in pki folder, postgres-server.crt , postgres-client.crt in pki/issued folder and postgres-server.key and postgres-client.key in pki/private folder.   PostgreSQL JDBC library cannot read .key file, which is why we have to convert the key to DER format (.pk8) file.   openssl pkcs8 -topk8 -outform DER -in postgres-client.key -out postgres-client.key.pk8 -nocrypt    Give proper unix permissions to the certificates and keys, for eg.   # chown postgres:postgres postgres-server.key  # chown postgres:postgres postgres-server.crt # chmod go-r postgres-server.key     2. Edit postgresql.conf ssl = on          ssl_cert_file = '/opt/postgres-sec/postgres-server.crt...

OSM Data can be used for variety of usecases like rendering, searching, data analytics etc. Each usecase yield quickest result in a particular data format. Due to the large volume of data, this can cause significant difference in the response times.  Some of these formats are, PBF  -- A highly compressed format used almost exclusively for storage and downloads owing to the small sizes. These format are not human readable and gzip compressed. PostgreSQL with PostGIS extension -- A data format backed up by PostgreSQL database. There are various schemas supported for multiple use cases which can greatly impact the response time as indexes and normalization are used to speed up queries. Some popular schemas along with their usecase and certain features are as below. The table is taken directly from openstreetmap wiki. Schema name Created with Used by Primary use case Updatable Geometries ( PostGIS ) Lossless hstore  columns Database osm2pgsql osm2pgsql Mapnik ,  Kothic...

BACK UP EVERYTHING BEFORE TRYING THIS. The entire process can be split into 4 stages. 1. Physically increasing the vdi file size on the host machine. This requires help from the hypervisor OEM. For VDI, which is a native VirtualBox image, we'll use VBoxManage. VBoxmanage modifyhd centos.vdi --resize 100000 If the above command doesn't work and throw an error about this feature not being implemented, you might try the following instead. Create a new disk with higher capacity VBoxManage clonemedium --existing 2. Resize the LVM. For this we'll use GParted to resize the image. Boot up a GParted live CD and resize the required LVM partition to suit your need. 3. Resize the LVM-2 Boot up the machine with the newly extended image and check df -h and fdisk -l. You'll see that the disk has indeed grown, but LVM hasn't. Use the following command to resize LVM to full size. pvresize /dev/sda2 (assuming your LVM partition is sda...