I don't think it has anything to do with being a "rich guy's club." Being able to run a node shouldn't be a "rich guy's club" either. And though mining already is, do we want to exacerbate that? Maxblocksize is very much about preventing the externalities that limit node/miner participation into more and more centralized groups.
I think "artificial limit" implies a misconception. Bitcoin as a system was a matter of designing a system that optimally aligns market incentives to ensure a decentralized, secure network. We cannot have talk about an "artificial limit" without mention that this limit is specifically designed to ensure decentralization and security---the former by maximizing the ability to run nodes and participate in mining, and the latter by forcing users to pay some of the cost of confirmed transactions, rather than externalizing all costs onto nodes and miners (forcing some off the network). I take exception when people try to make this about "free markets vs. central planning."
Bitcoin was centrally planned from Day 1; the entire point is trying to design the optimal system, with regard for all of its parts. The point wasn't to disregard most of the system so we can guarantee cheap/free transactions for users.
blocksize doesn't affect miners as much as nodes. Miners who can afford to mine in this market can afford to have bigger blocks too, because blocksize is limited by nodes, not miners.
And 98% of nodes right now can support 4MB blocks without a problem.
Block size certainly affects miners greatly. An entire block needs to be relayed across all nodes in a timely manner for mining to be decentralized. The more data propagated, the more latency.
Where do you get this 98% number? The IC3 paper suggested that 4MB blocks would kick 10% of nodes off the network, and they only tested propagation effects.