Without moderation you would still heavily influence it as you are heavily trusted so whatever trust list you use will be very heavily weighted through the community.

If you want a less centralized system then I would suggest removing yourself from the accounts that is trusted from DefaultTrust, choosing people who wish to remain reputable throughout the community (and are very trustworthy) to be trusted by DefaultTrust, and to only remove people from DefaultTrust under very specific circumstances that are outlined publicly and are open to public debate (not vote - shills).

A very good example of when a reputable person who is trusted by DefaultTrust acted when the community was concerned about his trust report is here. We should strive to have more users like philipma1957 trusted by DefaultTrust (who care about their reputation and are willing to act on their trust list when someone calls out an issue about it)