As far as I can see, no stable market equilibrium can be reached by all miners at the same time[/b]. For mining market has the peculiarity that whenever a miner increases its own supply, the supply of all the other will decrease. In contrast to regular markets where the players only compete to meet the demand, Bitcoin miners also compete to increase their own supply at the cost of their competitors since total block production remains capped even with unlimited block size.
I posted the problem of unlimited block sizes in another forum and got an interesting response from CubicEarth:
@alkan You make some good points about how a change in one miner's orphan rate necessarily changes the orphan rates of their competitors. I don't agree, however, with your conclusion that "It will all end up in a doom loop.".
First, regarding the lack of a static equilibrium, it's true that miners will need to continually optimize their blocks in response to ever changing real-world conditions if they are to maximize their profit. This is expected though, and as much is true of any entity involved in production of just about any product. Second, even as a theoretical exercise, if it is fact not a solvable equation, that still doesn't mean there will be a run-away effect. To my reasoning the optimal block size for any given miner may oscillate back and forth around some mean, or collectively it could look something like an n-body orbital simulation, which despite the appearance of some occasional chaos, as a system remains symmetrically anchored around a center of mass, and adheres to the conservation of energy.
In your examples, you make no mention of the mempool demand curve, which is how much each successive unconfirmed transaction is will to pay as a fee per byte. The fees are what the miners are after, and the orphan rate is a concession they make in pursuit of those fees, and is otherwise just something to be minimize. It's simple to think about: why wouldn't A, B and C all produce giant blocks? Well, potentially if there were enough transactions that paid high fees, the blocks could be large, but in the absence of that, any miner could stand to gain an advantage by making their blocks smaller, and having a lower orphan rate.
I really like the comparison with the n-body problem and now even think that the block size could in fact stabilize around an equilibrium point. It seems that there must be a block size (the central mass) where no individual miner has an incentive to deviate by building smaller or larger blocks. This equilibrium wouldn't lie at the intersection of the market demand and supply curves (which cannot be just summed up as in regular economics since they are mutually dependent). Thus the block size chosen by the miners wouldn't directly maximize their revenue as a group but minimize the risk that any individual miner could gain an additional profit by deviating from the block size. I may be wrong on this though.
What do you think?