Letting miners _freely_ expand blocks is a bit like asking the foxes to guard the hen-house— it's almost equivalent to no limit at all. Individual miners have every incentive to put as much fee paying transactions in their own blocks as they can (esp. presuming propagation delays are resolved or the miner has a lot of hashpower and so propagation hardly matters)— because they only need verify once the cost of a few more cpus or disks isn't a big deal. In theory (I say because miners have been bad with software updates), they can afford the operating costs of fixing things like integer overflows that arise with larger blocks, especially since they usually have little customization— other nodes, not so much?

Since miners can always put fewer transactions in, it's not unreasonable for the block-chain to coordinate that soft-limit (— in the hopes that baking in the conspiracy discourages worse ones from forming). But in that case it shouldn't be based on the actual size, but instead on an explicit limit, so that expressing your honest belief that the network would be better off with smaller blocks is not at odds with maximizing your revenue in this block.

If you want to talk about new limit-agreement mechanisms, I think that txout creation is more interesting to limit than the size directly though... or potentially both.

Even for these uses— Median might not be the right metric, however— consider that recently it would have giver control to effectively a single party, at the moment it would effectively give it to two parties. You could easily use the median for lowering the limit and (say) the 25th percentile for raising it though... though even thats somewhat sloppy because having more than half the hashrate in your conspiracy means you can have all the hashrate if you really want it.