It was a good idea on paper. But it needs to be moderated. Otherwise, trusted people will be able to give negative feedbacks for no reason to people they dislike, which is unfair.... You could be able to appeal a trust.
The trust system is just a way of aggregating people's opinion on each other. There should be no appeal process since my opinion about you can't really be appealed. If I hate your guts and think you are scum, then I have a right to that opinion. If you find out that someone is just giving negative feedbacks for no reason to people he hates, then remove him from your trust list and it won't affect you one bit!
In short, you decide who is on your trust list, and if you don't want to trust people who are loud about their opinion and you think they have no good reason to be giving negative feedback, then remove him from your trust list and problem solved!