Isn't this true only if you know their respective public keys? Because if you don't, you have no better method than just brute-forcing against the 160-bit hash.