I'm not talking about "bad actors".  The fact that someone has a different goal than you doesn't necessarily mean that your goal is "good" and theirs is "bad".

The point is that a "Consensus Building Process" can't work if participants goals don't align, especially if the participants are passionate about their goals.

Sure you can reduce the influence that you allow to those that disagree with the majority, but that isn't "consensus", that's "majority wins" democracy. It's just another form of "exile".  Instead of immediately rejecting their input, you slowly remove their input until their input becomes meaningless.