If a campaign manager doesn't accept you into a campaign any longer because you distrusted them, they are abusing their power. I would be curious to see if you have evidence of this happening or if this is just a fear you have?
There is a manager who has more priority for users who have custom trust list. This indirectly indicates that "put me in your trust list".