The funny thing about statists is that the first argument they use when anarchists complain about taxes is "just move, then," but when property use comes into play, that argument is never brought out.
The owner cannot set whatever rules they want. Their property will still border the property of others and those others will not be willing to accept certain levels of risk that the private owner is willing to put on them. Calling the property private does not solve the problem of risk management.
Then they can move.
Or they can simply join together as a community and force the offending property owner out, or otherwise impose their will upon him. Which is what they already do, and so we've come full circle.