I'm wondering, with high variance on p2pool or another somewhat smaller pool, is it possible to actually lose money? Like in the long run luck is irrelevant, but what if you have really bad luck and then the difficulty goes up 30%?
In this situation do you lose money because of high variance?
Yes you can. Consequently you can also gain money. Contrary to what people will insist on, it does NOT even out in the end because difficulty keeps rising. However luck determines if you're ahead long term or behind long term in an irretrievable way. If diff did not continually rise and rose-and-fell or stayed the same it would even out.
Your argument is based on the statement that difficulty keeps rising. While historically this is true, it is not a guarantee of future patterns. Yes, we have seen some insane rises in difficulty, but that is due to a number of factors including wider acceptance, general knowledge of crypto-currencies, incredible hardware advances, etc. I don't think we can state as a fact that difficulty will forever continuously rise.
I did not argue that it will rise indefinitely. However, it's realistic to predict it will continue to rise until it levels off. Once it levels off, it is never going to fluctuate as much as it did to pre-rise diff levels. Until it levels off (and this could be years away) this issue will hold true, and all the variance experienced till that point can never be caught up on. As I said though it's not a guaranteed loss, it's almost as likely a guaranteed win. The fact that by definition bad luck means blocks take longer means the statistical bias is towards a loss, but that statistical bias is nowhere near as large as the variance effect itself is.
Put it this way, many diffs ago, on a lucky day in mining, you could earn 10 bitcoin in one day or 1 bitcoin on a bad day with 60GH. In today's diffs, even if they fluctuate a little instead of just going up, there is no way in an eternity you could ever make up those 9 bitcoin with 60GH unless you used your avalon as a weapon to hit someone over the head and steal their money.
The next block halving will probably be the only time we'll ever see a significant drop again (unless BTC value crashes). What effect that will have on a tight margin network (unlike the GPU days) is unpredictable.