The block number/height is not a part of the block, it is not used anywhere inside the block, when miners are hashing blocks they don't specify something like "we are now mining block n+1, they are mining a random block.

What changes in the process is that in order to mine the next block you need the hash of the previous block, not the number of the previous block.

To explain further, you can assume all miners mine empty blocks just to keep the transaction out of it, so miner A now has this data that he is hashing in order to solve a block

Previous block hash + data + nonce
Let's say it is

4444 + 123 +1 so hashing goes on with a different nonce

4444+123+1
4444+123+2
4444+123+3

Lets say at this point miner B hit their own block, they would inform miner A, miner is here isn't going to start form scratch, it is not like he lost the race, he would just start hasing using the new block hash

5555+123+1

At any given second here miner A can hit a block, or 5 in a row, miner A's ability to hit block is directly related to how many hashes he can perform, it has nothing to do with how many blocks miner B can find or how "fast" miner B does find blocks.

I am probably not good in explaining things, but I think it gets clearer when you really understand that blocks are found by brute force randomness, most online articles say it's solving complex equations, that gives the illusion that all miners are trying to solve the same puzzle, but they are not.