Can they collude? Assuming they did collude the only thing they could do is block transactions and it would be instantly detected by all peers. A fork would be launched immediately with these delegates stripped and the two chains would compete against each other in the market. Assuming the delegates were just random people that attempted to screw with the system then consensus would be maintained.
What happens if somebody sets up 51 delegates, provides incentives to get the votes, and then takes over? Its much easier than trying to get 51% of the mining power.