However, with no limit on block size, it effectively becomes miners who are in control of _everyone_'s block size. As a non-miner, this is not something I want them to decide for me.
Only miners (and by that I mean solo miners + pool operators, not pooled miners) need to be a full node. They're your "everyone" there.
Please people, understand one thing:
you can't run a full payment network used by millions as a hobby. The day Bitcoin becomes something really serious (i.e., thousands of transactions per second), being a full node will necessarily be a professional task. It won't be something you can do on your laptop, "just for fun".
EDIT: And, as Mike said, the idea of converting Bitcoin into some replacement to SWIFT with $20 fees for transactions, which would force people to use bank-like institutions for daily transfers, just because you want "ordinary people to verify transactions", totally turns me off. Bitcoin can be much more than that. If you actually want it to remain this censorship-resistant currency that it is, it has to remain suitable for small transactions like buying some plugin from Wordpress. If you want Bitcoin to remain an alternative for those seeking financial privacy, you have to keep it suitable for SR users and alike - otherwise all these "bank-like" payment processor would ruin your privacy. If you want Bitcoin to remain an alternative for those trying to protect their purchasing power from inflation, you have to keep it suitable for those who want to protect their daily money on their own, without having to use a bank just for storage purposes which would recreate the incentive for fractional reserves. The list can go on. Bitcoin has the potential to be
much more than SWIFT 2.0. But for that, processing transactions will have to become a professional activity (it kinda already is actually).
But maybe not, and if just one miner starts creating gigabyte blocks, while all the rest agrees on 10 MiB blocks, ugly block-shunning rules will be necessary to avoid such blocks from filling everyone's hard drive (yes, larger block's slower relay will make them unlikely to be accepted, but it just requires one lucky fool to succeed...).
Succeed in what? Killing everybody else? Do you realize that would likely require more than 50% of the network processing power, otherwise the "unacceptably-gigantic" block would always be an orphan? Miners would likely reject blocks way too large, specially if it's filled with transactions never seen before (i.e., a likely attempt of flooding).
There is of course wide spectrum between "I can download the entire chain on my phone" and "Only 5 bank companies in the world can run a fully verifying node", but I think it's important that we choose what point in between there is acceptable.
I'm glad you see it's not so black-and-white. I'm sad though that you think you can actually "choose what point between there is acceptable". Such point cannot be found arbitrarily, even because it is not fixed: it varies all the time according to different demands for security, transaction space, speed etc, and different supply of resources. Basically, you'll never be able to even list all data that is necessary to take such decision, as it involves subjective opinions of thousands and eventually millions. Let alone calculate anything.
The curious task of economics is to demonstrate to men how little they really know about what they imagine they can design.

Then I think you misunderstand what a hard fork entails. The only way a hard fork can succeed is when _everyone_ agrees to it.
Only if you want to totally avoid the fork actually. Otherwise, you may have both chains living in parallel for some time. Then eventually one might "beat" the other, as people migrate. Or both keep alive "forever", that's a possibility too.
The more I see these discussions, the less I believe in totally avoiding the fork.
EDIT: And also, as a general comment on the discussion, you people fearing "too much centralization", as in "too few market participants", should realize that, at most, what would happen would be a few pool operators, like we have now. Pool operators do not own the processing power. Such processing power will remain scattered among thousands of people, who may easily migrate to different pools if they feel like. Pretty much like what already happens. Current pools need to have some "professional bandwidth" if only for protecting against DDoS, It already require professional resources to run a mining pool.