Any new ideas on this?
When I do a local search to find matching addresses, it doesn't really matter much for the search time whether it's 1 or a million addresses. I'm pretty sure using a database would be much faster to get one address, but I don't know if it can beat the speed of searching a text-file for a million matches.
TXT search speed is ok. THe problem is downloading the database. I would like to avoid that. I will try to figure something out.
I tried with rsync, but even an incremental download updating a day-old sorted file requires downloading more than a GB. I would have expected it to be less, considering most lines are unchanged.
It would be really cool if you can just enter a million addresses and get the balances (okay, yesterday's balances) quickly after uploading the 40 MB list.
I believe the only way to do that would be to use my own node to get the data.
For now, I am using blockchain data from third party nodes (mempool.space and blockcypher). And both of them have limits. After some requests you get blocked, and neither of them allow you to request multiple addresses at once.
Do you still have a server running a full node?
I never worked with a full node, but I can try to make an API for it that would allow such requests.