I considered that actually, but I determined that it was more efficient this way and could see no adverse problems. Could you expand on your concern?
What frequency do the challenges arrive? They might be too infrequent to deter an attack, but if you require them with each vote then they are synchronised exactly with the control they affect and they make the attack get very expensive very quickly.