I understand the block limit is an anti spam measure,
but why can't miners just form their own consensus of
how big a block is acceptable?  If more than 51% of the
mining power agrees a block is too big, they will ignore
it and build a longer chain.

We might have more reorgs but what's wrong with this
idea?

The main idea behind this is that actually, having a blocksize
limit was a temporary measure and in fact should not
be a protocol rule at all.  That's why we are having so much
difficulty.  We are trying to form consensus on implementation,
rather than the core rules.