If all the non-mining nodes reject the block, then the vast majority of the network doesn't have a copy of it. It basically isn't a real block.
The network worked before there were ever non-mining nodes having a copy. So if only the miners have a copy of the transactions then it is more than good enough.
Your example is like a straw poll. The non-mining nodes had no real authority to actually enforce their conclusion.
Like I said the only vote that holds authority is if you win a block, then your vote on the previous block validity counts as a confirmation.
If you mine an invalid block your vote isnt counted by anyone except you...everyone else ignores you, and probably blocks your node if you do it enough. And it still isnt a vote.