Or they can simply join together as a community and force the offending property owner out, or otherwise impose their will upon him. Which is what they already do, and so we've come full circle.
Might makes right, huh?
Nope, but the community has a right to defend itself against an individual that would choose to put them at risk.
In other words, Might makes right. The community is not a person. It has no rights. If the individuals who feel they are at risk don't want to move, they can take it up peacefully with the property owner. If they use violence to push him out, they are not defending themselves, they are aggressing against him.
Our opinions differ in this matter. You won't succeed in convincing me, I won't succeed in convincing you. Further discussion will serve no useful purpose. It has been an interesting discussion nonetheless.