So, unless you have 51% of the computing power, you can't decide which fork is the valid one.
That's a feature, not a bug.
No, I'm saying that even with 51+% of the computing power, you CAN'T decide which fork is the valid one if the wizards don't agree.
Take for example the current situation.
"If you are using Bitcoin-Qt/bitcoind version 0.7.2 or earlier, you must take action before 15 May, 2013. If you do nothing, you are likely to be left behind and will be out of sync with the rest of the Bitcoin network."
Notice it doesn't say that if 51% of the computing power decides not to upgrade... never mind.
It says, we've decided. Come along or not.
[edit]
So the significance of what I'm saying is that, if a handful of people already make these decisions. And the rest to the bitcoin users already trust these people to make these kind of decisions, THEN
there are trivial ways to create a distributed consensus among a handful of known individuals.
It doesn't require an ever growing amount of computing power.