SegWit have no downsides while increasing the blocksize could give us centralization to a certain degree as the blockchain size increases over the years.
any kind of change will always have both downsides and upsides. so it is wrong to say SegWit didn't have any downsides! because it did. for example the fact that it is opt-in makes it an inefficient block size increase. and also the complexity of it has been a problem for services to adopt it.