Obviously, there would be exceptions, and this is where communication comes in. Ostracism is the preferred method of punishment for most crimes among the majority of anarchists I have encountered, and we are rather extreme about it. You make restitution for your crimes, or you literally do not get to interact with ANYONE. With proper communication, this can spread for a rather large, if not global, distance. It provides a rather strong incentive for someone who has transgressed against his neighbor to settle the issue.
So you implicitly suggest that all people share the same principles of life and moral values. It may be true for a small isolated group of people and your idea of ostracism and outlawry may actually work between them. But if you take some larger proportion of population and apply this principle among them you will see tensions arise and eventually you will end up with fractions denying and neglecting each other (if not fighting)...
The emboldened text is a biggie. It's beyond logic to keep such assumptions, that's why they're always implied rather than stated. There was a guy who made an entire nation dream of a world without commies, Jews and faggots. How did man's inborn moral sense, the natural grasp of wrongs and rights, play into this?
What's interesting is the level of abstraction in these debates. "The Government" is always assumed to be a distinct, immutable entity, separate from the ones being governed. That allows a for a "bad guy" to point a finger at, but in reality it's a continuum -- from the president to the street sweeper on the government payroll. The IRS agents also pay taxes -- it's a tangled mess. There's simply no "other" to kill with fire here.