So now that I have my private key in Electrum, I can set the fee to what I want. Looking at https://estimatefee.com/ if I am happy with to confirm within 144 blocks ( ~1440 min), I can pay 5 satoshis/byte which will work out around $4.79.

Is there any reason why it would be a bad thing to let my confirmation take this long? I'm transferring between my own wallets, so it's not like trust is an issue.