What wallet do you use? Most nowadays use a dynamic fee to help you avoid this. When on mycelium I know if I choose an "economic" fee option it's going to take 3+ hours to confirm. If your wallet doesn't use dynamic fees (where the fee changes based on where it should be for that day) you should switch wallets or go to the website I linked above to see what a decent fee would be. They change all the time so you can never count on one certain fee an any given day.