Exactly. You need three elements:

1) A wrongful act, that is, one that violates a right.

2) A harm, that is, provable damages.

3) A connection between the wrongful act and the harm.

Trespassing on property is a wrongful act. Physical damage to property is a harm. Property rights and not just the right not to have your property harmed, they're a right to not have your property trespassed on or messed with without your consent. Physical damage to property is not the only harm that can result from a trespass.

There are tons of exceptions to these rules though, there have to be or you get very seriously crazy result in weird cases. Some of those exceptions are absolutely required to have a sensible legal system and some of them are judgment calls that reasonable people can disagree on. For example, I could see a Libertarian society making an exception to the requirement of provable harm in the case where the wrongful act is a trespass that violates a significant privacy right.