IP can more easily be debunked from this prospective:
An non-violent act (copying information), needs a violent act (the enforcement of IP rights) to be stopped.
This isn't true at all. I'm not sure what kind of libertarian/voluntaryist you are, but there are many voluntary ways to have IP. First of all, IP can simply be enforced through a DRO mechanism if the society recognizes it as protected property (which is just as arbitrary as real property if you think about it). Even if the society doesn't accept it, I could have it so that before I give my book to you, I have you agree to a contract that you will not copy it and sell it or give it away or distribute in any fashion, and again I can use the DRO if you violate this.
There is an ontological difference between genuine property (people, land, things like hard drives containing bitcoin blockchain patterns) and those things referred to as "intellectual property" (ideas, patterns and concepts). Even physical things which we refer to as "superabundant" like water, air and sunbeams are actually "scarce goods". There is some limit to their quantity even if it isn't practical to treat them as such a lot of the time. There is no conceivable limit to things like ideas and patterns though, besides those we would derive from limits in the actual world of scarcity. I could write "I love you." an infinite amount of times up until I die of old age or run out of paper or pixels.
This is the way the world is, so no it is not just some arbitrary distinction between IP and real property rights. When property rights are assigned to ideas and patterns, this aspect of the legal order creates a conflict within society. If you look at the history of IP, you will see that things like patents have been, from day one, state grants of monopoly. Even though a person copying the pattern of words in a book doesn't deprive an author of any of his own scarce things, a state gives an author license to go take from "infringers".
I don't think that much of the gamut of IP law will survive the transition to a free society besides law related to fraud. In some really high value situations like an employee charged with keeping a copy of a movie secret before release to theaters, it might be expected for this employee to actually agree to the type of contract you propose. If a movie company depends in large part upon being able to release the movie to theaters first and get revenue before the high quality, easily copied DVD version is released, they might be able to get employees to agree to a situation where the consequences of letting a DVD lay around and end up getting copied are severe.
There's nothing wrong with people agreeing to this type of arrangement, but when A and B make a contract, there is no obligation on C to not copy a book. Your hypothetical free market IP has no teeth beyond A and B, the only people who actually voluntarily agreed to the terms. Therefore it would be necessary for A to force B to agree to some sort of harsh punishment for not sufficiently protecting A's secret word patterns. It just seems unlikely to me that people would agree to this type of contract on a $20 book, when the market will provide an equally good book for $25, or else the law will be just so hard to enforce profitably like things are today and some unique work will just end up pirated.
Your statement makes about as much sense as "An non-violent act (squatting on someone's homestead), needs a violent act (the enforcement of property rights) to be stopped." This is a pretty solved problem in the voluntaryist community.
If I got drunk, wandered in to your house and took a nap on your sofa, would that be a property rights violation IYO? I really don't think I should have to explain why this type of thing is "a violent act", or that if you argue this point anyone should take anything you say seriously.