decision to force a fee market is a centralized solution
On it's face this is a nonsense argument since any development decisions are centralized in the same manner.
Increase the blocksize, decrease the blocksize, or leave it alone, they are all (centralized) development decisions.
It's also false that anything is really centralized about it because if there were truly a consensus for change (over the objections of the 'centralized' developers) there would be a successful fork.
Yes all dev decisions are essentially centralized, including the decision to NOT do something. Since that is trivially true, I am talking about the effect of the decision. And in one case miners can optimize their profitability by choosing to include transactions while in another case they are artificially limited.
Listen to New Liberty, he got this completely right. Whether miners can optimize
their profitability is beside the point, because in doing so they also influence
others' costs, and they are most certainly
not optimizing that.
The idea of a sensible market arising for block size in the current structure if the consensus block size rule (which is the only mechanism for the "others" in the previous paragraph to participate in such a market) is a fantasy.
You are correct to question whether the incentives for miners coincides with what is optimal for the rest of the network. As we see with empty blocks and non-validated blocks these can differ. However, your assumption that these do not coincide whatsoever is unsupported. And with Bitcoin it is all we have, barring some centralized committee making arbitrary decisions like we have today with the core devs. Personally, I believe that miner incentive is a reflection of optimal network use, if not 100% the same. For example, a miner who mines a bunch of "spam" txns knows that he must store it for eternity, just like all the other nodes.
A better example is the worry that a miner will include a txn that takes 10+ minutes to validate. It is dangerous to build on a block that can't be readily validated. So a miner that includes one could expect that the majority hashing power would NOT move to that block, even though it may be the longest in the chain. The "fittest" miner will mine 2 blocks in the time it takes others to validate this one and mine the next. That need only happen once or twice to teach the rest of the network a valuable lesson.
So miners will evolve to be tweaking their algorithms to eliminate "crazy junk" (unless that junk is sufficiently incentivized with a large txn fee -- and if someone is willing to pay a lot for it, who are we to decide that its not important... that's the free market at work) and even include "human assist" systems to optimize decisions like these. We may get a few "natural" forks longer than just one or 2 blocks as miners choose different strategies but that is just part of the natural behavior of the core consensus algorithm. All of this can be done without changing consensus.