Quote from: Desmond1543 on February 02, 2019, 09:15:30 PM

Damn I wish I was more fluent in programming. Right now I am using the blockchain.info API to check balances, which is totaly retarded. As you said, I need to check all used addresses on the blockchain instead.
I was reading about crawling and saving all addresses from the blockchain? But there are over 300 million of those, so yes, I would need a bloom filter (as I understand it, it basically reduces size?)

So there are two approaches, either crawl the entire blockchain, and search through it every time.
Or crawl it to find the public key so I dont have to search through a database.

First, you have to look only at the UTXO set, you don't care about addresses used in the past and now empties

Updated at block # 547944 30/10/2018

output # addresses Tot bitcoin

P2PKH 18.453.794 10.541.332
P2SH 3.865.985 4.906.667
P2PK 38.678 1.759.927
P2WPKH 62.643 126.738
P2WSH 18.662 11.812
MULTISIG 1-1 357 0.056
MULTISIG 1-2 142.354 23.24
MULTISIG 1-3 205.226 17.85

TOT 22.787.699 17.346.536

so we are talking about 20 million, not 300.

With bloom filter the searching time becomes the neglegible part. If you have a list of all addresses (not base58 encoded) in UTXO

addresses.hex

Code:

0000000000000000000000000000000000000000
0000000000000000000000000000000000000001
0000000000000000000000000000000000000002
0000000000000000000000000000000000000003
0000000000000000000000000000000000000004
0000000000000000000000000000000000000005
0000000000000000000000000000000000000006
0000000000000000000000000000000000000007
0000000000000000000000000000000000000008
000000000000000000000000000000000000000a
0000000000000000000000000000000000000011
000000000000000000000000000000000000001a
0000000000000000000000000000000000000023
0000000000000000000000000000000000000064
0000000000000000000000000000000000000092
0000000000000000000000000000000000000100
0000000000000000000000000000000000000246
0000000000000000000000000000000000000258
000000000000000000000000000000000000028f
00000000000000000000000000000000000002fe
.......................................

you can use the program I linked to get from that list a 512 MB bloom filter called funds_h160.blf (addresses with funds)

First you have to download the program (I assume you use Linux) and compile it:

you can download it from https://github.com/ryancdotorg/brainflayer/archive/master.zip and unzip or use the 'git' command:

Code:

git clone https://github.com/ryancdotorg/brainflayer.git
cd brainflayer/
make

then run it

Code:

./hex2blf addresses.hex funds_h160.blf

[*] Initializing bloom filter...
[*] Loading hash160s from 'addresses.hex' 100.0%
[*] Loaded 18503292 hashes, false positive rate: ~2.162e-22 (1 in ~4.625e+21)
[*] Writing bloom filter to 'funds_h160.blf'...
[+] Success!

To perform a search in the bloom filter, suppose you have a list of 1000 addresses generated from 1000 private keys to check against the bloom filter:

Code:

for (uint16_t i = 0; i < 1000; i++){

   if (bloom_chk_hash160(bloom, addresses[i])) { //if there is a match between addresses[i] and one of the addresses in bloom filter

   printf("Found! Key number %08x\n" , i);
   exit();

   }
}

where the bloom_chk_hash160 function is defined here -> https://github.com/ryancdotorg/brainflayer/blob/master/bloom.h