Thanks for that, it was my reading also.
Thus TX fees that are not in the block but paid out of band are not subject to penalty...
It's an additive deduction, not multiplicative.
You seem to be thinking the miner's reward is:
(1 - penalty) * (minted coins + tx fees + collection from pool)
Where it is really:
minted coins + tx fees + collection from pool - penalty
Having more fees in the txs included doesn't increase the penalty. There is no difference between adding 1 mBTC to the fee and paying 1 mBTC out-of-band.
I don't see how this differs from in Monero?
In Monero, addition of txs up to the median blocksize is free. As you surpass the median blocksize, a quadratic penalty is applied to the subsidy of the coinbase, but amounts obtained from tx fees are untouched. The subsidy of the coinbase is initially dependent of the number of coins in existence, and so takes into account the previous penalties to coinbases of any previously generated blocks (comparable to your "pool" method). Then, the miner is free to add transactions meeting some economic equilibrium that maximizes their overall income when taking into account the blocksize penalty to the coinbase subsidy.
So, it's like this:
(1 - penalty) * (minted coins) + tx fees
where penalty is dependent on the size of the block above the median size according to the formulas found in the CN whitepaper.
gmaxwell criticizes this as promoting out-of-band transactions, but the fact remains that to permanently and secure transfer money you must use the blockchain and have it included in a block somewhere. Thus, I never thought it was much of an issue.