So then they'd have to constantly downvote me on almost every post, and if you were to check the history, you'd see that it very much looks like I'm getting targeted even tho my posts might not be deserving of that mass downvoting. So once again, it can be detected very easily and it would require them a lot of work to target some guy and downvote him.
They won't care whether it's detected or not, they will just do that because you busted them for some wrongdoing and at that point they don't have anything to loose. And problem with your suggestion is that it doesn't have any mechanism to prevent abuse, which will happen for sure. I've seen it many times, scammers leaving negative feedback to those that busted them, but since they get excluded from other DT members their feedback is irrelevant.
What's stopping some high ranking member to leave a negative trust and say something like:" excessive spamming" or something like that without the ref post or any proof. Same thing would happen, as in your suggestion.
Leaving negative feedback for spamming is not the proper use of trust system. If someone abuses that constantly chances are he will get excluded from DT. I don't know if you read i before, but I suggest you to check
LoyceV's Beginners guide to correct use of the Trust system. Even theymos said that LoyceV guide seems
reasonable.