I'm wondering, with high variance on p2pool or another somewhat smaller pool, is it possible to actually lose money? Like in the long run luck is irrelevant, but what if you have really bad luck and then the difficulty goes up 30%?
In this situation do you lose money because of high variance?
Yes you can. Consequently you can also gain money. Contrary to what people will insist on, it does NOT even out in the end because difficulty keeps rising. However luck determines if you're ahead long term or behind long term in an irretrievable way. If diff did not continually rise and rose-and-fell or stayed the same it would even out.
Your argument is based on the statement that difficulty keeps rising. While historically this is true, it is not a guarantee of future patterns. Yes, we have seen some insane rises in difficulty, but that is due to a number of factors including wider acceptance, general knowledge of crypto-currencies, incredible hardware advances, etc. I don't think we can state as a fact that difficulty will forever continuously rise.
Just as you can have swings of bad luck, you can also have swings of good luck, which you correctly argue. It would be an interesting experiment to see where the "tipping point" is. In other words if we assume X change in difficulty, what is the point Y of bad luck from which you will likely never be able to recover?
There is a forum member named davejh who does some really great statistical analysis. I'd love to see him plot this kind of thing.