Why not always go to the pool with the lowest number of shares? It would seem (at first blush) that this would result the highest returns overall. Intuitively this would seem to be the best way to capture disproportionately large shares.
There are not only proportional pools involved. Slush has a scoring method, bitcoinpool has a fine if you leave too early, etc. So it's more complicated to calculate. I guess if all pools would be proportional and have the same fee structure that approach would work.
Also, adding smaller pools could work if your server could pull equally from the X lowest pools where X could be configured based on the overall hashrate of the metapool. I understand the concern about these pools dying, but the flip side is that contributions from the metapool may sustain more of them, since the pool owners really only care to make their fee.
I agree. Most pools publish their cumulative hash rate. So there could be a limitation like "don't put in more than an additional 50% of their usual hashrate". That doesn't solve the problem though if you choose a pool which has no reputation and runs away with the BCN...