I actually was under the impression that we are libertarians here and we should push for any solution that decreases centralization.
No block size limit = free market decides the proper size
You mean the 3 people controlling the pools that have the majority of hashing power control the blocksize.
Actually, no. Individual miners have freedom to choose a pool which suits their interests better or simply use the P2P pool.
OR start a new pool and convince enough people to use it.
Indeed you can, but you've moved away from the standard Bitcoin model where we hold miners hostage by saying if they do something we dislike, we'll ignore the blocks they mine. If we were having a discussion about what the inflation subsidy would be, no-one would ever suggest putting it to a miner vote; as Jeff Garzik said a few months ago the decision is very much like the inflation subsidy:
http://garzikrants.blogspot.ca/2013/02/bitcoin-block-size-thoughts.html (he proposes a fixed blockchain growth rule, although that has many problems of it's own because you can't predict the future)