Why don't we just let miners to decide the optimal block size?
If a miner is generating an 1-GB block and it is just too big for other miners, other miners may simply drop it. It will just stop anyone from generating 1-GB blocks because that will become orphans anyway. An equilibrium will be reached and the block space is still scarce.
Unfortunately it's not that simple for a couple reasons.
First, right now
clients will reject oversized blocks from miners. Other miners aren't the only ones who need to store the blocks, all full nodes do even if they just want to transact without mining. So what if all the miners are fine with the 1-GB block and none of the clients nodes are? Total mess. Miners are minting coins only other miners recognize, and as far as clients are concerned the network hash rate has just plummeted.
Second, right now we have a very clear method for determining the "true" blockchain. It's the valid chain with the most work. "Most work" is easily verified, everyone will agree. "Valid" is also easily tested with unambiguous rules, and everyone will agree. Miners can't "simply drop" blocks they don't like. Maybe if that block is at depth -1 from the current block, sure. But what if someone publishes a 1GB block, then someone else publishes a 1MB block on top of that? Do you ignore both? How far back do you go to start your own chain and try to orphan that whole over-size branch?
I think you can see the mess this would create. The bitcoin network needs to operate with nearly unanimous consensus.
I know the 1MB is a hard limit which affects both miners and clients. I'm assuming a world without MAX_BLOCK_SIZE at all, both miners and clients.
Miners can ALWAYS drop a valid block if they don't like it, just like ignoring any valid transaction. Currently, miners taking non-standard transaction has higher risks of orphaned block because other miners may not like these block.
If a miner (Bob) sees a new valid block with height N but doesn't like it for whatever reason, he will simply keep mining on top of block N-1. When Bob finds another valid block (N
2), he will broadcast to the network and other miners will choose one between N and N
2. Here Bob takes a risk of being orphaned because other miners may build on block N. If block N+1 is built on N, Bob has to reconsider the risk and he may decide to keep mining on N+1, instead of N-1 or his N
2. However, if Bob (or his team) owns 51% of the network, he will always win and block N must be eventually orphaned. (You may call it a 51% attack but this is exactly how the system works)
Therefore, if the majority of miners do not like 1GB block, building 1GB block will become very risky and no one will do so.