I don't really think that the people getting neutral feedbacks would care much about it as long as it is not affecting their account from the outside. You might find only a few in thousands who would take that feedback as a warning and stop doing what he got it for.
That's the problem, Neutral feedbacks sometimes were left unnoticed. Since it somehow not reflecting on the trust column, which in results to the continuity of an act which they thought is right.
On the other side, leaving a negative feedback to someone not clearly proven guilty for abusing the system sounds unfair too. So the better solution as said by a few already, would be to tag only the ones who are clear threats for the system, and have been constantly abusing it.
A proper process must be done and thorough investigation is a must before tagging someone, I must agree that leaving a negative feedback without a hard evidence to support a claim is somehow improper and inhumane.
A lot of users here wants to help in cleaning and clearing this community from account farmers and abuser/s. There's already a thread that in which some users can post users who're attempting to abuse the merit system. It would be a good tool or reference to use to conduct an investigation.