As far as I understand, the reason median is used is because it guarantees that the lower bound never goes backwards. With each new block found it either advances, or doesn't change. If the mean was used, then there is the possibility that the minimum timestamp could move backwards.

Thanks to an off-by-one error, not only is the last block of one epoch different to the first block of the next epoch, but actually the time it took to mine the first block of every epoch is not taken in to account whatsoever when calculating the difficulty.