[quoting Satoshi] An attacker can only try to change one of his own transactions to take back
money he recently spent"
The problem is that we won't know which transactions are theirs, so every transaction becomes suspect even if only a small number of transactions are directly affected.