Can they collude? Assuming they did collude the only thing they could do is block transactions and it would be instantly detected by all peers. A fork would be launched immediately with these delegates stripped and the two chains would compete against each other in the market. Assuming the delegates were just random people that attempted to screw with the system then consensus would be maintained.
What happens if somebody sets up 51 delegates, provides incentives to get the votes, and then takes over? Its much easier than trying to get 51% of the mining power.
First of all someone with 51% of the delegates only has one power: to exclude transactions or to stop producing blocks.
If they gain 51% of the delegates and fail to include transactions that would vote them out then it will be very obvious to everyone on the network, the blockchain will be hard-forked by some developer and the network will continue to function. You see what binds a network is not the software rules, but the social contract among the community. It is almost inconceivable that in the event 51% of the delegates stop including transactions that the community would sit back and take it for long.
Competition among chains and no barriers to entry or forking keeps people honest because there is NOTHING to gain by a 51% attack.
Note that these 51 delegates would have to pretend not to see blocks from the other 49 delegates and thus the attack would be so obvious and disruptive that it wouldn't last more than 24 hours before community members took action.