There can be minor delays in the time but block times generally don't get longer because even if some miners stop mining, the blocks will still be mined by others.
If some miners suddenly stop mining, the average block time will increase until we have the next difficulty adjustment.
Once the difficulty is adjusted (assuming the total hash rate hasn't been decreased by more than 75%), the average block time will become 10 minutes again.