Or they can simply join together as a community and force the offending property owner out, or otherwise impose their will upon him. Which is what they already do, and so we've come full circle.
Might makes right, huh?
Nope, but the community has a right to defend itself against an individual that would choose to put them at risk.
In other words, Might makes right. The community is not a person. It has no rights. If the individuals who feel they are at risk don't want to move, they can take it up peacefully with the property owner. If they use violence to push him out, they are not defending themselves, they are aggressing against him.