I know how you feel... The fees there are high. I made a 0.002 BTC transaction and had to paid at least 11.000 satoshis fee with the estimated complete time in 60 minutes. I tried to reduce more the fee, but it didn't let me, said the fee was very low and insufficient to complete the transaction. So, we don't have a way to pay lower fees... 
The fees recommended by blockchain is always based on the total size of your transaction. Higher transaction size = higher fees, so it is not based on how much btc you are going to send. If you want lower fee then your transaction will be on medium priority. Perhaps you should move to xapo wallet if you want to avoid fees.
Xapo is more complicated to use. How do I change the transaction priority to medium for an example?
I usually don't want fast transactions, because I don't need it, I just want to pay lower fees.
Why would you need to pay lower fees with Xapo when Xapo pays the fee on your behalf? It doesn't have an option to prioritize your transaction unlike blockchain as Xapo is like a bank and not just a wallet.