I think Erdogan's insight into the game theory behind the block size limit was correct.
But to recognize his insight, I think we need to stop thinking in terms of "valid blocks" and start thinking in terms of "valid transactions." All blocks that are composed exclusively of valid transactions are valid.
Now instead of thinking that only Core and XT exist, imagine that there are dozens (and in the future possibly hundreds) of competing implementations of Bitcoin. Each implementation has its own rules for what block size it will build upon. From this viewpoint, the "effective limit" is the size of the largest block that's ever been included in the Blockchain. If a miner wants to create a larger block (e.g., to collect more fees), then he has to weigh the chances that his block is orphaned with his desire to create a larger block. If we imagine that the block size limit across the network forms some distribution as shown in the chart labelled "NEW THINKING" below, then, since the miner can't be 100% sure what this distribution is, it is rational for him to use the tip-toe method to minimize risk.
I think this is confusing a protocol enforced "limit" and market preferences.
There can not be disagreement on the protocol enforced limit, which is what your right hand graph shows. If there was then miners that issued larger blocks will get forked off of every miner with a lower limit.
I think the current situation is also more representative of the right hand side graph, than the left. Today all miners have a fixed protocol limit of 1MB, but many have preferences for smaller blocks. For example the stress tests showed just how many still had the 750KB soft limit in place. So we in practice have the right hand graph today.
What is needed instead is to get rid of the protocol limit in practice (maybe keep a high water anti-spam limit which is what the 1MB was/is), while letting the market show it's preferences. This would be like a combination of the two graphs where an anti-spam limit is far off to the right of the graph, and below that miners show a range of preferences on block sizes they are willing to both issue and accept, which looks like your graph on the right.
This situation probably leads to a loose and dynamic form of market consensus on sizes. Miners that decided to only accept blocks well below most other miners' preference risk being orphaned at a higher rate and so are forced to up the size they accept to better match other miners. At the same time miners that issue blocks larger than what most other miners are willing to build on also risk being orphaned at a higher rate. The result is miners are forced by market pressures to move towards a consensus.
This is where we should be and the hard protocol limit prevents the market from properly functioning.
What "many" have preferences for smaller blocks specifically? Who?