Is it simply because of how default fees are set in wallets, by exchanges, or by services? If so, why do professional developers miscalculate the fee so poorly?
They do not miscalculate, not always at least. In the case of exchanges, they either set a priority fee to ensure the transaction gets confirmed on time or use a very high fee so they could profit by keeping some of it (it's most likely the former in this case).
Some non custodial wallet as well also pre determine the fee range, electrum mobile app for example doesn't allow users set the exact fee they wish to use but give a range from low priority all the way up, so it's not possible to go below a certain amount.
As long as there are people willing to pay higher than the minimum amount, services would increase their default fees so as to stay ahead of the crowd; much better users complain about the Bitcoin network than their service.