An oligopoly is a situation where there are very high barriers to entry.
Exactly. For mining itself, 1MiB blocks ensure that the barriers to entry are very low, and thus an oligopoly of miners won't form.
Increasing the block size, and especially allowing miners themselves to determine by how much, increases the barriers to entry, allowing for an oligopoly to form.
Bitcoin itself may be an oligopoly, but we really, really do not want Bitcoin mining to become an oligopoly.