How do you figure? Does not associating or wanting to associate with people do violence to them? Am I aggressed against if someone doesn't invite me to a party? Ask me to marry her? Hire me? Give me a loan?

If so, then you seem to have a different understanding of "aggression" than me. I'm talking about initiating physical violence or threat of violence, and I'd like to hear someone make an argument defending doing that.
However, if you agree that that type of aggression is bad, then anything else is changing the subject.