If you trust someone (you add them on your trust network) then their 'opinions' will matter.
If you don't trust them(remove from your trust setting) and whatever 'opinion' they might have wouldn't matter.
Except that the offenders are not in most people's trust network because many people have the offenders in their trust list (actually very few people have the offenders in their trust list). The offenders are in most people's trust networks because a single person added the offenders to their trust list.
Even if some people exclude the offenders from their trust network (which does appear to be happening), the standard remains to be the Default Trust network.
The trust system was designed to work like a market based system, however the way it has been implemented has resulting in the trust system being far from market based.