But there isn't a changing difficulty for finding the private key as there is only ever 1 answer.
I see. To defend against this attack, people could just regularly transfer their savings between different wallets. If the target private key doesn't stand a good chance of containing enough bitcoins, the attack is kind of pointless, no?