How many times have dictators have used that exact same argument to take the rights of minorities? "If you don't listen to them and you don't let them speak to you, you deny them a vector from which to infect other minds." Just replace "them" with the minority group of your choice (muslims, jews, homosexuals, etc.) and you see how dangerous this is.
People choose to be hateful fascists. The situations aren't nearly similar. You can't just say "Yeah, but if the situation was completely different it would be bad", because the exact point is that it's this situation.
There's a difference between going "You're bigoted and cruel, and I refuse to support you." and saying "You have this arbitrary trait, so I hate you."