The limit determines the size of the consensus network.
If nodes cannot keep up, they will be pushed out.
On one hand, operators of full nodes have the right to decide what the limit should be, on the other hand, there must be a single static limit agreed upon via consensus. Otherwise more important full nodes (large miners, exchanges, merchants) might want to increase their limit via user-definable option and begin pushing home-based full nodes out of the consensus.
8Mb is a reasonable middle point for the next 4 years.
Your post shows exceptional logic and clarity.... until the last sentence.
We want to stay as close as possible to actual network demand and there are very reasonable opinions as to why blocks should get filled on average until we precipitate any change.