The same argument can be made against a contract. One who breaches a contract is not initiating violence or making threats on your property.
A person binds themselves to the terms of a contract voluntarily. That isn't the case with intellectual property. Even if you make everyone you sell a CD to sign a contract that says "I will not redistribute the contents of this CD", you still haven't recreated intellectual property because there is nothing stopping a third party that didn't sign that contract from doing so. A contract is binding because it's voluntarily agreed to. You can't automatically bind third parties which is why intellectual property isn't legitimate.
I completely agree that it doesn't make any sense, just like piracy doesn't require that you are causing me physical harm, but rather take away FUTURE revenue from my mental work.
That's because you've been deluded into thinking that you own the products of your work. That's false though. If you steal from me a slab of marble and then chisel a statue out of it, do you own the statue now? No, the statue is mine and I also claim that you've destroyed my slab of marble and owe me for damages. Once you empty your head of that delusion, you will stop feeling entitled to own an idea just because you thought it up.