The fear is that a cartel of big, centralized, have-huge-data-pipes miners would drive out smaller miners by forcing up the block size high enough so the smaller miners have to drop out.
Sounds like an irrational fear to me. Big centralized miners couldn't care less about little miners. Little miners are insignificant. And what are these big centralized miners going to use to fill up these huge blocks? Fake transactions? That makes no sense. Real transactions? Then there's a real need for such huge block sizes.