Those who have received negative ratings from me should, if they understand the Trust system and its use, be wholly unconcerned if my accusations are baseless, and vocally critical if they are true.
Being in the default trust network means your feedback is artificially given higher weight than others, so yes it is concerning if the accusations are baseless.
Finally -- and I do not presume to speak on behalf of CITM -- but his rating of me is based on past dealings between the two of us and it would be an abuse of the trust system for him to remove it because he feels I am abusing the trust system

Removing someone from his trust list is not abusing anything.