Is it simply because of how default fees are set in wallets, by exchanges, or by services? If so, why do professional developers miscalculate the fee so poorly?
It's pretty much this. I always check out how congested the mempool is before deciding on a fee.
As for the developers miscalculating the fees, I think it's mostly the case that they intentionally significantly increase the fees for UX purposes. Because the last thing they want is people complaining why their transaction isn't confirmed yet even if the wallet indicated that the tx would be confirmed in an hour or so.