Everyone is free to trust or distrust anyone they please for any reason they wish. Such is the nature of the trust system. It's impossible not to start collecting exclusions if you've been around for a few months and are active in Meta or Reputation - something you say will eventually rub someone up the wrong way, even if you didn't intend it to. It's not worth stressing over.

I also want to briefly mention that no reasonable person should consider someone giving merits to someone else as those users having a connection, endorsing what each other say, or even agreeing with each other. Merit should be seen as "This post is constructive/worthwhile/required a lot of effort", and not "I like this post or I like the person making it". Your other context is correct, of course, and I trust users who are distrusted by other users on my trust list, and I distrust users who are trusted by other users on my trust list.

I would also suggest that although everyone is obviously free to trust or distrust anyone they please as I stated above, distrusting someone solely as retaliation because they distrusted you is not a mature way to build a trust list.