Yeah, it's not that you don't lose an amount of mining, if you are off for 10 hours then you expect to lose 10 hours worth of mining, but it's that it is spread out over 5Nd so the variance effect is VERY low vs what your reward should be minus 10 hours.

If instead the N was only 1 hour, the variance effect is actually VERY high, i.e. your chance of getting less or more than your expected reward, is a lot higher.
So as long as your hardware DOES fail a REAL lot on that other pool, it's expected to even out OK eventually.
If being offline is a rare event, then you may win some and may lose some, but if you are in the 50% that loses, you don't expect to get it back.

But again, it's yet another one of those things that a lot of people on other pools don't notice or don't understand