Post
Topic
Board Project Development
Re: [Project Showcase] spruned - Bitcoin Light Client & Full Node Emulator
by
coindax
on 09/04/2019, 08:15:27 UTC
Hi,
it's because in first place I used the electrum network to download the block headers, and this implementation is still in the codebase today.

The electrum network uses the jsonrpc serialization, so the data is transmitted as an hex string. Citing the Electrum protocol page:
<<... now past height 500,000, the headers form over 40MB of raw data which becomes 80MB if downloaded as text from Electrum servers.>>
This explains why the bandwidth usage is higher than it could be (You didn't asked explicitly, but I think it was part of a complete answer :-))

The design fault here is, instead, to store the the blockhash as hex in an sqlite table.
A spruned instance will so have an headers db as the following, for mainnet:
142,172,160  headers.db
Because of the related indexes (by hash and height).

(On the leveldb side, developed months later I did, instead, a better job avoiding unefficient data serialization).

It's in development a larger usage of the P2P network and a refactor of the repository to further reduce the bandwidth and storage usage, that will lead to a smaller database.
Changing the repos, however, implies to add a migration system, which is something that I'm working on carefully. Alembic would be a good and easy choice, but I'm trying to use new dependencies as less as possible.

API are quite stable today and I don't see a lot of effort, in the upcoming releases, on creating new features.
Now is exactly the time to polish this kind of naive implementations and increase efficiency, before adding more stuff.