Unfortunately over-eager increases of the soft-limit have denied us the opportunity to learn from experience under congestion and the motivation to create tools and optimize software to deal with congestion (fee-replacement, micropayment hubs, etc).
Probably the best time to let soft-limits persist was in 2011/12 when the ecosystem was smaller, the funds at stake were a lot smaller, users considered the software more experimental than beta, and the world's press wasn't really watching.
Look at the huge abundance of space wasting uncompressed keys (it requires ~ one line of code to compress a bitcoin pubkey) on the network to get an idea of how little pressure there exists to optimize use of the blockchain public-good right now.
My experience of (centralized) financial systems over many years is that ignoring hardware and software constraints as they are approached invariably causes outages. Also, that trying to train a user-base or worse, a market, to behave differently to accommodate IT constraints is a Sisyphean task. There are probably hundreds of IT experts who are concerned about the block size limit, because they can see the risks in it, which they recognize from prior (usually bitter) experience.
And, this is where the role of Core Dev is crucial. If there are major efficiencies to be had, "low-hanging fruit", then it would be wonderful to see them go live and reflected in smaller blocks etc. But right now, we can only project forwards, from what is happening with the average block size.