Why don't we just let miners to decide the optimal block size?
If a miner is generating an 1-GB block and it is just too big for other miners, other miners may simply drop it. It will just stop anyone from generating 1-GB blocks because that will become orphans anyway. An equilibrium will be reached and the block space is still scarce.
Unfortunately it's not that simple for a couple reasons.
First, right now
clients will reject oversized blocks from miners. Other miners aren't the only ones who need to store the blocks, all full nodes do even if they just want to transact without mining. So what if all the miners are fine with the 1-GB block and none of the clients nodes are? Total mess. Miners are minting coins only other miners recognize, and as far as clients are concerned the network hash rate has just plummeted.
Second, right now we have a very clear method for determining the "true" blockchain. It's the valid chain with the most work. "Most work" is easily verified, everyone will agree. "Valid" is also easily tested with unambiguous rules, and everyone will agree. Miners can't "simply drop" blocks they don't like. Maybe if that block is at depth -1 from the current block, sure. But what if someone publishes a 1GB block, then someone else publishes a 1MB block on top of that? Do you ignore both? How far back do you go to start your own chain and try to orphan that whole over-size branch?
I think you can see the mess this would create. The bitcoin network needs to operate with nearly unanimous consensus.