Why not use a fixed time going backwards like the forwards time?
Because then that time would change. If the rules said a block could not be mined more than 2 hours behind network time, and I mine a block with a timestamp 1:59:59 behind current network time, then by the time I broadcast it to the network, some nodes will start to reject it since it will now be greater than 2 hours behind network time by the time they receive it. By using a median of the last 11 blocks, the lower limit for the timestamp is fixed before the block is mined.
The longer we go without a block the larger the backwards time range becomes, a fixed time provides a fixed range from past to future blocks.
Note that we use the median, not the mean. The median of the last 11 blocks is the timestamp of the 6th last block. It doesn't matter if one of the blocks took 1 second or 1 hour to mine - the median will not change.