There would be a dispute system. If an issue arises, the two must make their cases. It will then be up to the public to vote based on the stories told.
So a person like Quickseller who would make multiple voting accounts could scam freely? How would you "vet the voters" without manual investigation?
They will have to put some stake behind their votes. They couldn't just vote. They would be required to put some money behind each vote. If they lose they will lose the amount of currency they put in. But that's just one side of the equation.
There will be a feedback system as well. So there will be some preventative measures taken by the user to make sure that they do a good job in practice.
Won't work - scammers will find flaws in any automated system.