As I understand, the 20 XRP serves as the payment for the space in the ledger. You can't withdraw it. No matter what, the information/transactions that have been made with that account was already stored in the ledger and you pay for it even if the account will not be used any longer.

Visit the link, a good discussion and explanation about the issue : Fully remove all XRP from an account.