You have to understand where he's coming from. He and the Bitcoin Foundation are pushing Bitcoin as a system to do payments on the internet; the recent San Jose conference had the tagline "The future of payments" after all.

It's a lot harder to convince people that investing time and money into implementing Bitcoin for payments is a good idea with a 1MB limit on transactions. Removing the blocksize limit entirely and making it something that miners decide solves that problem from that perspective: regardless of what the demand from transactions are at least one entity will always be able to meet that demand at a cost approaching the cost of bandwidth and servers. That's why Gavin likes to talk a lot about "free market forces" and "competition" when it comes to mining, and has said before he's happy to see the smallest 20% or so of miners and full-node operators get forced out of business by rising costs every year.

From the perspective of someone who wants to accept Bitcoin payments on their online store letting the majority of miners decide what the blocksize is solves the uncertainty of how much transactions will cost. They connect directly to miners to send their transactions and don't care if Bitcoin is controlled by six people or six million. From the perspective of someone investing in Bitcoins because they want a decentralized store-of-value, AKA electronic gold... well they might see things differently.