I don't understand how your algorithm will give the entity with higher hashpower more profit by mining blocks that give less BTC per block, that doesn't make any sense to me. Why wouldn't the bigger miner mine smaller blocks and get the higher BTC per block, like your example was showing?
On the other hand, if it really is more profitable to mine larger blocks, then why wouldn't the smaller hashpower nodes do it? If they do, then the same problems I mentioned before would apply (The penalty would give a competitive advantage to the higher hashpower nodes).
if this is true;
Reward per block for 1% miner: 5943 * 0.0006885 + 3.2223 - 0.4589 = 6.8552 BTC (more than in scenario 1)
Reward per block for 90% miner: 7251 * 0.0006885 + 3.2223 - 3.5294 = 4.68521 BTC (less than 1% miners in this scenario; more than the miners in scenario 1).
Why isn't the 90% miner simply including 5943 transactions like the 1% miner is, in order to get about 2 BTC more per block?
At some point, the increase in penalty outweighs the increase in tx fees + collection, so he stops there. Whether one or more of these summands is more than 2 BTC is irrelevant, it's their sum that counts.
The problem is, regardless of what the collection is in the pool, the highest profit per block is always greater by simply mining at the target block size.
If the major nodes have driven up the BTC in the collection pool to provide greater profits on the whole, then the first big miner that stops contributing to the pool will essentially drain the pool by mining blocks at the target block size. In the end, trying to maximize profits by relying your competitors to "play fair" won't work.
This is actually analogous to cartelization; Big companies cooperating to achieve greater profits. However.. Cartels, absent the power of the State, always break down, because the first one that does can achieve greater profits at the expense of the cartel as a whole.
Weaknesses of cartels
Analysis demonstrates that a cartel is an inherently unstable form of operation:
If pooling resources is more profitable, then the cartel will merge into one company.
If it proves to be less profitable, the individual members of the cartel will break off.
If it doesnt break from within, an outsider, noticing the enormous profitability, will enter the market, and this dooms the cartel.
Particularly likely to be restive under the imposed joint action will be the more efficient producers, who will be eager to expand their business rather than be fettered by shackles and quotas to provide shelter for their less efficient competitors. If the cartel does not break up from within, it is even more likely to do so from without. To the extent that it has earned unusual monopoly profits, outside firms and outside producers will enter the same field of production.
http://wiki.mises.org/wiki/Cartel