Under these circumstances, telling me that I should "keep your hands off of that which isn't yours" is imposing your morality on me.
The alternative is you imposing your morality. Since you're okay with that, you should have no objection to other people "imposing their morality" by telling you to butt out. In your view, they are just as justified as you and you have no way to justify imposing yourself. In their view, you are clearly in the wrong, and you still have no way to justify imposing yourself.