The potential for abuse without a block-size limit is too high, if the block size limit is removed, then it will be the end of all chains that adopted the removal.
Correct. This would only work if all participants of the system were honest and mean no harm. This is obviously not the case with Bitcoin as it keeps being attacked via various methods (FUD campaigns, spam, TX spam, etc.)
It is however, far too early for this to happen. What is needed right now is a dynamic block size limit, that expands using a predictive model of the same function which models the minimum increase in network bandwidth over time around the world.
There was talk about this being implemented at a later point. However, one has to set up very good 'limitations' so that the dynamic block size limit can't (hopefully) be abused in any way.