Sure, but where is the limit? Can you cause harm, based upon the belief that your neighbor intends to do harm to youself, your property, or your family; in order to prevent your neighbor from doing harm? Can you do the same, if your neighbor intends you no harm, but you believe that he is a threat to others? How can you make such a determination?
I'll give you a tip here... you need to stop saying "you" and start saying "society". Because it's not about ME and YOU, it's about US ALL. It's not whether I alone believe you to be a threat, it's whether society as a whole believes you to be a threat.
When you play the me and you game, it's easy to shoot down the regulation system because it looks more arbitrary.
Gotta give you an A for subtle, manipulative effort though!