You're counting each private key as 5,000,000 comparisons, when in my mind at least, it should be 1 comparison. So you're effectively doing 10,000 comparisons every 4 seconds, unless I'm missing something. Just load the 5,000,000 addresses into memory, it'll be fine.
Here, something I developed a while ago. It may help:
https://crates.io/crates/utxo-scannerhttps://github.com/mdizak/rust-utxo-scanner