I say it matters because my property right is the right to keep my property in the place and the configuration I prefer (provided I have the right to occupy that place). Plain and simple, no weaselly subtleties. It's not about the value or even the utility of my property: it's about the physical matter it's made of, and my control over where it's located and how it's arranged.
So if I break into your house (without damaging anything), go through all your stuff, and put everything back the way it was, that doesn't violate any of your property rights?
That concept of property derives from the nature of physical matter, which is only able to be in one place at a time. You can't drive my car to New York at the same time I want to drive it to Los Angeles. You can't blow it up while I keep it intact. It's one or the other, so we need some means to decide who gets their way. Our society has chosen to do that by assigning an owner to the car and letting the owner decide where it will be. Since I'm the owner, you're violating my property rights if you move or alter the car against my will; the damages I suffer are approximately equal to the cost of restoring it to the way I want it.
I sneak into a private museum without paying the required fee. Nothing wrong with that?
Rights have to have some source. There has to be some wrongful act or some negligence or you have to meet one of the narrow special cases where this is not required (strict liability). If you can kill with legal blows, that's fair.
This seems circular. How do we know whether an act is "wrongful" without knowing whether it violates rights?
It is wrongful if it violates rights. If I can damage your property without violating your rights, that's tough.
Say I shine a flashlight in your window. That might or might not violate your rights, most people would say it doesn't. Say I shine a flashlight in your window and ruin some film that you had developing. The ruining of the film is the damage you suffered, actionable if and only if shining the flashlight in your window violated your rights.
You seem to have this backwards. If my violation of your rights damages your property, that's actionable. But damaging your property alone is not a violation of your rights. You have no right not to have another's actions damage your property. Otherwise, if I drove my car down my driveway and created vibrations that ruined your glass structure, that'd be actionable even though driving my car down my driveway is something I'm supposed to have the right to do.
Your property right means you have the right to say what can and cannot be done with your property, subject to other people's rights. (For example, if your ball lands in my yard. Or if you want to use your baseball bat to break my nose.) One type of damage you might suffer is physical damage to your property. But there are other types of damages. In theory, a violation of any right could result in any type of damage. There's no law that says that the only damage that can result from a violation of property rights is physical damage to property.
Say you and I both have a shot at a particular client. It's near certain the client will pick you or me. I disconnect a wire in your car. Your car doesn't start. But there's no damage, reconnecting the wire costs nothing. If you miss the meeting and lose your chance at the client, is that actionable? I violated your property rights not by damaging your car but by trespassing on it. The damages aren't the physical changes to the car but the meeting you missed. I violated your rights, you have damages fairly attributable to that violation.