I considered that actually, but I determined that it was more efficient this way and could see no adverse problems. Could you expand on your concern?
What frequency do the challenges arrive? They might be too infrequent to deter an attack, but if you require them with each vote then they are synchronised exactly with the control they affect and they make the attack get very expensive very quickly.
The challenges are constant and will not span voting sessions. At minimum you'll receive at least 1 challenge per voting session. If you are connected to a number of nodes, then you'll get at least 1 challenge from each connected node for that voting session (you'll also be sending challenges to the same connections).
In your 10,000 example above, you'd have to process at least 10,000 challenges per voting session, and you still only get 1 vote in that session per "node".
Each voting session will generally last between 30-60 seconds depending on whats happening network wide and if there are any conflicts.