Why wouldn't miners reject interactions with miners who set the block size too high, for instance?
Yes, I believe they would. So far, most miners and pools are VERY conservative; I think the idea that they will create huge blocks that have a significant risk of being rejected, just so they MIGHT get an advantage over marginal miners that can't process them fast enough, is loony.
Rejections are is
not a significant risk for miners. That's the whole point of my original post on the issue. If your blocks are built upon by the majority of hashing power, you come out ahead in the long run. Your orphan rate does increase proportionally, but if, say, 5% of the hashing power never sees your blocks, the increase in varience is low, yet you get the very real benefit of 5% less competition. In the long run there is no "might" - it's simple statistics, and I haven't seen anyone offer a rebuttal based on analysis rather than hand-waving.
I just don't see why widespread norms wouldn't emerge to have the blocksize be within a certain reasonable range at any given time (the norms would change dynamically/organically as the mining community sees fit), with the result that rogue miners would be more or less
completely isolated.
Tragedies of the commons only happen if there is a forcible ban on anyone taking ownership or control of any part of the commons. Since each node is in control of itself, at least, I don't see how this situation can be subject to a tragedy of the commons. It seems that if any behavior was know to be harmful it would become taboo and miners would understand that to protect their long-term investment they had better reject rogue behavior.
Isn't the debate then a matter of how many people would reject? My sense based in economic reasoning suggests that norms like this could cover much more than 5%.