Xapo is a shitty online wallet anyway,  so why not switch to a better one?  The fee you described is pretty ridiculous,  but even mycelium requires that much even for the 'economic' option,  which works just fine.  This whole argument is why altcoins are starting to look better every day.