Yes, because for one thing the village probably hasn't elected the "band of thieves" democratically and "robbing" is per definition violent, which collecting taxes usually is not.
As has been pointed out, taxes
are violent. They're always backed up by the threat of violence from the state.
And democracy is itself founded on a logical fallacy: The idea that if a lot of people agree on something, it must be right.
So supporting taxation is, essentially, the idea that if a lot of people think it's OK, you can commit violence on people, even those that disagree.
Tell me, if slavery were democratically approved, would you support it? Even if you were part of the minority chosen?