Post
Topic
Board Development & Technical Discussion
Re: Goxplorer, a Bitcoin blockchain parser package written in golang
by
iMil
on 29/05/2020, 04:16:50 UTC
Version 0.8.1 has landed.

This release's main focus is HTTP mode "real life" usability.  In this mode, it is highly recommended to convert LevelDB databases (block index and chainstate) to a more concurrent-friendly database, Badger https://github.com/dgraph-io/badger. This is done in one step, using goxplorer:
Code:
$ goxplorer -addr mkdb
It takes about 30 minutes to process all of the databases, YMMV depending on your hardware.

From there on, goxplorer will use the Badger databases instead of LevelDB. Of course, it needs to be updated regularly in order to keep in sync with the ongoing blockchain, and you have to take care not to use your node real databases as LevelDB only supports one process at a time, and this could corrupt your running database, leading to a reindexation. Here's a safe way of re-syncing Badger with LevelDB databases using an overlay filesystem https://wiki.archlinux.org/index.php/Overlay_filesystem:
Code:
#!/bin/sh

PATH=${PATH}:${HOME}/bin

DESTDIR=${HOME}/api
BTCDIR=/data/.bitcoin
OVERLAY=${DESTDIR}/overlay
export BTCBLOCKSHOME=${BTCDIR}/blocks
export BTCBLOCKINDEX=${OVERLAY}/bitcoin/blocks/index
export BTCCHAINSTATE=${OVERLAY}/bitcoin/chainstate
export BTCADDRDB=${DESTDIR}/addrdb

sudo umount ${OVERLAY}/bitcoin
rm -rf ${OVERLAY}/upper/*
sudo mount ${OVERLAY}/bitcoin
leveldbctl -d ${BTCBLOCKINDEX} i
leveldbctl -d ${BTCCHAINSTATE} i

curl http://localhost:7554/mkdb/
leveldbctl https://github.com/yuuichi-fujioka/go-leveldbctl is a very handy tool I've contributed to in order to make it hexadecimal-friendly.

This script can be for example coupled with a filesystem event detection tool like inotify-wait in order to be launched on every blockchain change, and it can be safely called while goxporer is serving HTTP requests.

* Homepage: https://imil.net/goxplorer/
* Releases: https://gitlab.com/iMil/goxplorer/-/releases
* Source code: https://gitlab.com/iMil/goxplorer