It just occurred to me: Removing the limit opens up bitcoin network for cheap DOS attacks.
How so? I believe I've explained pretty comprehensively why that shouldn't be the case: The attacker can just create non-op_return outputs, and those are even more expensive to process for the network because they go into the utxo set.