The universe is full of coercive forces, both non-human and human. The only way to remove other coercive human forces from your life is to completely remove all other humans. Otherwise in some way, directly or indirectly your life will be affected by other humans. No Ideology is beyond this simple rule......
If all humans were on a level playing field, without monopolies on force, everything would be quite alright. The potential coercive forces would have little relevance.
Except all humans being on a level playing field implies much more than simply a lack of a monopoly on force. Force, power, whatever you want to call it, will inevitably consolidate to some degree in any group because it gives an advantage to those doing the consolidating.
Even if you're right that a completely level playing field benefits everyone the most, there is significant incentive to the individual to make the playing field as unlevel in his favor as possible. In an anarchist society, I'm not sure how you can possibly prevent that from happening.