So, I send you 25 BTC, wait for you to accept it and send me my physical item, then I spend the same UTXO to myself and both are destroyed by your new rule, giving me my money back. This is exactly the same as a regular double spend.
You would lose your money, not get it back.
UTXO A (mine) = 25 BTC
send A (mine) to B (mine)
send B (mine) to you (C, yours)
send B (mine) to A (mine)
B and C are destroyed, leaving A, the original 25 BTC belonging to me