No wonders that third persons view this that way. But, for me, why should I bear something which is not true?
You can leave the same neutral feedback if you don't agree. This will clear up the fact that the neutral tag was wrong and provide a link to that thread.
To anyone who will be interested in your situation, you allow understanding of whether there was a mistake or not.
Only those who leave them can remove tags, if this cannot be done, you will simply have to ignore it.