Do you really completely get rid of your wallets once you're done using them?
Unfortunate, but keep in mind Satoshi designed addresses in the optic that they should be only used once. Therefore, it wouldn't make much sense to send the change to the sending address.