If I remember correctly, the time stamp of a block comes from the average of all of the miners.
If there were only two miners, they could use any forward time stamp that they'd want.
And as gmaxwell said, assuming the average of the miners is correct, the block numbers becomes a near indisputable time stamp for
the transactions that come on chain.. with enough miners and within a small error.
Block chain is one of the few things that can give a near indisputable time stamp.
It doesn't. We consider the median timestamp of the last 11 blocks, and not all of the miners.
There is also the point about network adjusted time, which prevents the timestamp from drifting uncontrollably; nodes wouldn't accept your block if your timestamp deviates too far from the network adjusted time. Even if there were two miners, it doesn't mean that they can set their timestamp arbitrarily. The blockchain doesn't necessarily provides for an indisputable or concrete "timestamp" but rather it provides an environment where you are required to expend far more to re organize the blocks.