Have you tried the "ignore" function? With some discipline, I am able to remove low-IQ contributors, whilst keeping most of those who have something to contribute - even if I totally disagree with it.
But the invite only/read only subsection is not a bad idea either.