My mental roadblock (or rather what cleared it) was the fact that even in a lopsided distribution you can still have an average, but it will be very skewed toward the thick end of the plot. I needed to actually see it plotted before it clicked.
This is why I stuck through, I thought I might help someone understand it better

Thank you very much for the follow up. I actually learned something too (at first I thought it is always wrong to calculate the times in advance, which, of course, given the correct distribution in times between finding a share, it isn't).
Although there is still one phenomenon I cannot explain. I have a HD 7950 and a HD 5830. The HD 7950 consistently gets 2-5% rejects, while the HD 5830 gets 6-12% rejects, even if I set clock and intensity on the HD 7950 such that the hashrates are equal, so it isn't dependent on hash rates.