Decentralization is not an important goal in and of itself. The end goal is censorship resistance.
User adoption contributes to censorship resistance. Therefore to the extent a block size limit restricts user adoption, it also diminishes censorship resistance.
But the correlation is 99% between centralization and censorship. Everyone things that by censoring only few evil transactions is good, but then soon find out that everyone will be censored.
You either have no censorship, or full censorship, there is no middle ground for it, it's a black & white issue.