Quote from: DeathAndTaxes on June 12, 2014, 04:45:00 PM

The sharechain doesn't contain the entire merkle tree and thus you can not verify what tx are being worked on by a share in the sharechain. Each share has to be as small as possible (to improve efficiency). It doesn't contain anything more than the bitcoin blockheader, the coinbase tx, and the merkle branch which allows cryptographic verification of the coinbase tx (linking coinbase tx to merkle root). Actually this is a simplification. It contains subsets of those elements because there are portions which can be independently computed by peers in the p2pool network and thus don't need to be relayed (i.e. the output of the coinbase tx). p2pool shares don't contain anything more than what is needed to prevent cheating by p2pool peers. For a p2pool share to contain all the tx hashes of the bitcoin block which would make them very heavy.

OK. It is important to keep in mind that even without any changes and without any pool providing any additional data, multiple pools using a p2pool-like backend would provide reduced variance. In order to prove that a given tx is being hashed by a given pool, there are various ways to go. The simplest would be for the pool to simply provide a merkle branch linking the tx to the merkle root. This may be a paid service.* Alternatively the pool may provide the full ordered list of tx hashes in the share. If the pool is also willing to provide the txs corresponding to any unknown hashes, this proves that it is not working on any private transactions that have not been disclosed to the rest of the network. This additional data (the merkle branch linking a given tx, or the full list of txs, need not be sent on the p2p network, but can be available from an API on the pool's server.)

Quote from: DeathAndTaxes on June 12, 2014, 04:45:00 PM

It may be possible to allow more efficient verification of "working tx set" but it would require a radical change to the Bitcoin block structure as well as block rules. If the merkle trees were generally similar or deterministic it would be possible to provide a merkle tree as well as a instruction set on how to select the txs. Like I said this is more useful for an altcoin as it would be a radical departure from Bitcoin.

I don't think any more efficiency is actually required. The additional data that is required to wholly or partly verify the "working tx set" need not be transmitted on the p2p network (in fact if the data is to be sold, then it cannot be sent on the p2p network), and is actually quite lightweight for something served via a traditional webservice.

Quote from: DeathAndTaxes on June 12, 2014, 04:45:00 PM

There is some merit outside of double spend proof for pools using p2pool as a backbone however there is also a tragedy of commons. Say p2pool represented 30% of the network hashrate this would mean any entity regardless of their actual contribution has the reduced variance of someone with 30% of the hashrate. Yes all participants benefit but they benefit to a lesser degree. The larger the participant the less they benefit and "worse" their advantage over other participants is reduced. Established pools are able to charge a fee because they are trusted and they reduce variance. By joining p2pool they undermine those two factors which are their sole reason for existence.

This is not quite correct. Larger pools in the superpool will still have lower variance than smaller pools, but it is possible for smaller pools in the superpool to have lower variance than large pools outside of it. A good example would be if BTC Guild and Discus Fish (both at 12% of network hashpower were to join into a superpool) Now BTC Guild has a lower variance, but it has come at the "cost" to them of also giving their competition Discus Fish a lower variance. Since they are equal in size and relatively small in comparison to the rest of the network, this seems like a good deal for both of them. If they were to let Eligius join their superpool, their own variance would get lower, and while Eligius variance would reduce, it would still be higher than theirs.

Quote from: DeathAndTaxes on June 12, 2014, 04:45:00 PM

To answer your question there is no technical reason why pools couldn't use p2pool (either the existing network or a private variant) as a backbone. There never has been. However there is also no "selfish" reason for them to do so. All the reasons relate to the "common good" which is a hard sell.

As I see it, the only pool that currently does not have an incentive to join such a superpool, is GHash.io. The tiny pools have nothing to lose, and the medium sized pools are better off joining a superpool with other medium sized pools (allthough possibly not with tiny pools) than not joining.

*This allows for new models of tx-fees. For example I could send a tx with no fees that would not normally get included in a block, but have a pay-per-share transaction fee contract in place with Discus fish. Discus fish would then work on including my tx in a block and receive a nanopayment from me for every share that they produce with a valid merkle branch linking my tx to the merkle root. The downside is that this introduces variance in my tx fee, I may get lucky and pay a very low tx fee if one of the first shares that mining provider produces is a block, but I may also overpay, significantly if it takes very long before a block is found. The upside is that as a consumer I can choose which pools I wish to support with my tx fees.