Exactly, thats the point. People do have the same kinds of weapons as your gangs, so everywhere a gang would form, a counter-gang would form, and the winning gang would surely be angry at the losing gang (especially if fatalities occurred), and the losing gang would flee to a surrounding area (or get killed off). Once they flee to a surrounding area, they will pick what they think is the winning gang there, and try to join it, resulting in each area being run by 1 gang constantly in fear of being taken over by its counter-gang (who they will naturally call "criminals") and surrounding gangs who have allied with "criminals."

And then the winning gangs become governments and we're back to square 1. The problem here is human nature: Humans like to form teams, and the winning team tends to punish the losing team, who then wishes to get revenge.

Decentralized violence begets more violence, and violence begets centralization. The only way of avoiding centralization is either reducing it just enough so that no violence breaks out,  but government is still sufficiently small, or by somehow getting rid of it entirely and replacing it with a system to prevent violence hands down.