We now live in the real world where there is IP.
You infringe, I find out, I send you a notice, you ignore it. You are now the agressor.
According to Libertarianism, no. According to the current laws, yes. The law used to be that you could own black people. I guess you would have supported that back then too, huh?