Transferring fund in different wallet would need to pay for transaction fee, you can not avoid that. You can only do is to lessen the transaction fee by using segwit address. Also i think you know what transaction fee is for, this would be a fee for miners to confirm your transaction and this is the basis of how long your transaction to be confirmed.