The limit determines the size of the consensus network.
If nodes cannot keep up, they will be pushed out.
On one hand, operators of full nodes have the right to decide what the limit should be, on the other hand, there must be a single static limit agreed upon via consensus. Otherwise more important full nodes (large miners, exchanges, merchants) might want to increase their limit via user-definable option and begin pushing home-based full nodes out of the consensus.
8Mb is a reasonable middle point for the next 4 years.
Your post shows exceptional logic and clarity.... until the last sentence.
We want to stay as close as possible to actual network demand and there are very reasonable opinions as to why blocks should get filled on average until we precipitate any change.
I appreciate the compliment.
Regarding the actual 8Mb figure.
The hassle of a hard fork (implied by the idea of a single static limit) should be enough to prevent individual nodes from meddling with the limit (via user-definable option or by recompiling the client), but because of the same reasons (the hassle of a hard fork) we don't want to do that every now and then. So we must future-proof it with regards to the competition and the current state of technology. That's why evolution has stages, it's not a continuous smooth process. You can look at it as a game of Chess as well, if you will. Now is Bitcoin's turn to make a move.
More on 8Mb justifications
here.