It will lead to even more centralization, if that is even possible. If there is a small pool, its variance further increases, and everyone would like to mine on the big pools.

With lower block times, the bigger pools also have a different advantage when getting to the next block as they don't have to worry about propagation time.