Just to make sure everything is clear, the penalty pool is not related to mining pools, despite the usage of the word "pool". I have not talked about mining pools in this discussion.
Sorry my mistake. I was thinking miners could leave mining pools in this scenario, to benefit from the remaining members of the pool producing larger blocks.
I decided to analyse the situation in the case of mining pools. As a start, I used Meni's parameters as described in
https://bitcointalk.org/index.php?topic=1078521.msg11557115#msg11557115.
Assume I have 1% of the total hash rate. I am currently in a 90% mining pool. There are also 10 1% miners. Using the results in Meni's post, my expected income per block is 0.04685 BTC.
Will I earn more by leaving the 90% mining pool and solo-mining?
If I solo-mine, the 90% mining pool is now an 89% mining pool and there are 11 1% miners. Solving for these parameters:
n0 = 7217 (slightly smaller)
n1 = 5945 (almost the same)
p = 0.6906 mBTC (slightly larger)
Penalty paid by 1% miners: f(5945) = 0.4602 BTC
Penalty paid by 89% miner: f(7217) = 3.3043 BTC
Average penalty: 0.89*3.3043 + 0.11*0.4602 = 2.9914 BTC
Reward per block for 1% miner: 5945 * 0.0006906 + 2.9914 - 0.4602 = 6.6368 BTC
Reward per block for 89% miner: 7217 * 0.0006906 + 2.9914 - 3.3043 = 4.6712 BTC
My expected income per block is now 0.06637 BTC, 42% higher than when I was in a mining pool. There is a very strong incentive to betray the large mining pool. I have not done any additional calculations, but I suspect it is also profitable for
every individual miner in the larger mining pool to leave and solo-mine, or at least join a smaller pool.
Thus a sensible large mining pool operator should not mine supersized blocks. Meni, I realise you've come to this conclusion another way (higher income means higher difficulty), but this is yet another reason why rollover penalties discourage large mining pools from mining large blocks.