Is it simply because of how default fees are set in wallets, by exchanges, or by services? If so, why do professional developers miscalculate the fee so poorly?
As for the developers miscalculating the fees, I think it's mostly the case that they intentionally significantly increase the fees for UX purposes. Because the last thing they want is people complaining(and blaming the wallet specifically) why their transaction isn't confirmed yet even if the wallet indicated that the tx would be confirmed in an hour or so.
I was about to comment this.
But I'll just add something: OP is in the pretext that the only variable in calculating the priority fee is the average mempool size.
In reality, clients consider more variables like how many block it took for a transaction in a specific fee range to get mined, etc.
The reason is it's hard to predict the network, they may be a surge of transactions after your transaction was broadcast making your "
1mb-from-tip" transaction become "
5-mb-from-tip" before the next block is mined.
It'll be better or worse than "
mempool estimation" depending on the implementation and network condition.
Given that, some estimation methods usually overpay when the average mempool size is high then became small in a very short period.