that is a good question. is it possible to speed that up, comparing two list with millions of lines?
If you sort the file and use binary search to look for each item in your list then your runtime becomes O(log2(n)) for each for each entry and so you're going to have a maximum of O(NumberOfAddresses*log2(n)) as your worst case runtime. It's really not slow, that's about 30 units of time to search for an address in a list with 1,000,000,000 lines in it.
Actually fitting all that into memory is going to be a problem though. There are some algorithms I read in a Knuth book about on-disk sorting but they're very old and I think they may have to be adapted for hard disks instead of tapes.
The original "brainflayer" tools set ( github ), had a tool called "binchk" included, where it used a sort-hex file to generate a binary file, so a 100GB hex file becomes a 5GB binary file, then binchk can operate on that bin file. Where you pass a list of hex addresses.
Binchk was written as a cover for hex2blf as the bloom filter is only 99.9999% positive, and if your doing 250M bitcoin addresses, that's 1,000'f of false positives, so you pass the false positives to binchk, where they're tossed out.
I'm seeing 1500M hashes/sec with each RTX-3070 on my rack, so with four I get 6,000M/sec, I have 300M used valid btc h160 addresses from stripping the blockchain.
Still as the man says, with 6billion/sec, on a bloom of 300M, the odds of a hit in a scope of 10^76, is quite small
But its even more clear that 'mining' btc is impossible, unless you have an asic farm and own a hydro-dam in china.
I don't use bitcrack, find it too slow, and poorly written GPU code. I used to run the 8gb bloom-filter on the gpu cards, but now I just have the GPU return batches of 1,000's of hex-addresses for check, then I can keep a 32gb bloom filter on the main desktop that is shared-memory.
I think that we're approaching 10,000M/sec pretty soon on high-end cards, with ram disks and 128GB ram, should be able to get the 64GB bloom down to false positive of 0.0000000000000001%
So you still need to do the binchk
I do think this approach is working, I watch the last of the last of public keys that have 50+ BTC, and there used to be +1000, now less than 900, and dropping by the week, of course we don't know how many low-value keys have been found, as there is no way to estimate loss.
When I last ran bitcrack I was only seeing a few 100M on a class 1070 card, its always best to write your own code, and know how it works.
I used to use the work of open-gl vanity-flayer base, but moved a few years ago to the jean-luc vanity-search approach, biggest advantage was seen in moving the bloom back to the CPU, but I think it has more to do with huge cheap memory now and ram disks. A real bottle neck to get 2,000G/sec off an RTX board is having the bloom filter on a super fast M.2 device that is only used for that purpose. Some of the new MOBO's have 3+ m.2 slots and each can have 8x 2TB
iMHO where this is all going is that once all the legacy bitcoin public-keys are GONE, that the writing on the wall will be kaput. If I can do this stuff, you can bet that kids all over India&China are doing same, some 3+ years ago I postulated that if these mining farms repurpose their gpu's to scanning hex addresses in used address space, they would earn more than they do from mining.
...
The reason I mention public-key is there are two approaches to hacking BTC, one of course is brute-force, just random create privates keys and see if they generate a high-val address.
The second approch is high-level math, where you pair the public-key of secp256k1 to a GF(n) near to secp256k1 order ( math term for number of uniq keys in field ), then you get a ball-park estimate for private-key on known-public key, then once you know the range, you use a tool like 'kangaroo' to find uniq private key for that public-key.
The best of both worlds is to hack 'kangaroo' to check the bloom-filters while it creating those 1000's of billions of random private keys per second.