Are you using Electrum's ETA fees estimation? That tends to be quite inaccurate as it does a floating average of the fees in the previous blocks rather than the actual mempool fees.

If you were to switch to mempool, you should be able to see much lower fees. You'd want to set it at about 1vMB for a confirmation within the next two blocks or so.