Post
Topic
Board Development & Technical Discussion
Merits 22 from 9 users
Topic OP
World's fastest and simplest block parser for those who (only) need all HASH160
by
BTCW
on 05/09/2023, 13:45:59 UTC
⭐ Merited by LoyceV (8) ,BlackHatCoiner (4) ,ETFbitcoin (3) ,JayJuanGee (2) ,mocacinno (1) ,citb0in (1) ,albert0bsd (1) ,DdmrDdmr (1) ,vapourminer (1)
The question never really dies; since Brainflayer in 2016, the dream seems to live forever, so to speak: "How can I get all HASH160 ever seen on the blockchain in one file?"

There are confusingly many different solutions here and on Github (and shadier places). One more advanced than the other. I went in the other direction, thinking, "It can't be that difficult."

It isn't. It's much easier than you think. Assuming you have a fully synced Bitcoin Core local installation (the .blk files on your hard drive, downloaded with txindex=1). Also, that you are on *nix (or have Ubuntu on a VM or something - more than good enough), this needs no extra downloads, no dependencies, not much extra HDD space, very little CPU and RAM, and no GPU.

Assuming you're in the terminal and the ~/blocks folder, just:

Code:
cat blk00000.dat| xxd -p | grep -oE '1976a914[0-9a-f]{40}88ac' | sed 's/1976a914//;s/88ac//' | sort -u > P2PKH-unique00000.txt

It shouldn't take longer than 1-2 seconds per .dat file, even on an old budget computer.

That's it. Done.

Some comments:

  • For all .dat files in one go, start with "cat blk* |" instead (and remove the numbers in the output file name). I opted for the first file only as the first proof of concept because there is zero waiting until you know it works.
  • What does it do? Briefly and on the fly - and in this order: Reads the blk file[ s ], changes format from binary data to human readable hexadecimal (the blockchain files have no deeper code to crack than that), searches for the P2PKH pattern, selects the "middle part," which is the HASH160, sorts all hits alphabetically and removes duplicates, forgets all else and writes the result to one text file.
  • Want P2SH-P2WPK and native Segwit too ("3" and "bc1q" addresses), maybe even Taproot? No problem, change the prefix and suffix bytes accordingly (very easy to find). I recommend saving these to separate text files since you can't tell "naked" HASH160:es from each other, but... whatever floats your boat.


Yeah, about the title. I think this is the world's fastest and simplest solution; I will, of course, change it when/if proven wrong.

Happy HASH160!