I recently activated my yubikey for blockchain.info.
So, I am wondering, if I would have fallen for that, would I have a problem?
In addition to the previous responder, if your attacker had access to an (encrypted) wallet backup then they woudl not need the YubiKey to open the wallet in Blockchain.info and coudl steal your monies!
You should keep (even encrypted) backup files very safe.