I think, for some, they simply look at what each pool can offer them for the month - not always looking at long term. I will use this snapshot as an example - wish that minergain went a full year back though to show a more thorough comparison, but for this example the past 2 months should work.

so in the example I will use 71 th/s (about average for 5 S9's) for the past 2 months, even with the increased payout due to smaller pool - I do realize that due to the ramping down of those that left that the payout would have been higher had they been fully ramped down - but for this example, this still works - with Kano finding 13 blocks, the reward would be $380 with Kano while it would have been $681 with slush and $755 with BTC
for some - that is all that matters - the monthly pay. My guess is because they often reinvest the profits to get more miners.
for the 3 months I mined with Kano, I used 5 S9's - I also had placed 5 S9's on 3 other larger pools. The net variance between the 4 different pools was very very minimal. So, yes the luck usually eventually even it all out. You may just need deep pockets to hold you over till it does, if you were not here when it was really good.