Because you can say that miners aren't contributing for the first hour of a block, or you could say they are helping to find the block in less than a hour. It's all relative, math, averages, etc. Play the averages, it's your best bet.
If going off averages was safe then the time based reduction in value of shares isn't needed! Its there because people can take advantage of the system.