And that is why non-aggression is so important.  You can do what you want as long as you don't interfere with others.