People worry about transaction fees being too low as a result of larger blocks, but isn't this a free market between miners and users?

If I am a miner, can't I create artificial scarcity by enforcing a soft limit? By simply not including transactions below my profitability threshold?

Similar to the milk producers pouring excess milk down the drain.

So maybe we should worry about the inverse actually: that miners will form cartels and start limiting even 1 MB blocks