If all the non-mining nodes reject the block, then the vast majority of the network doesn't have a copy of it. It basically isn't a real block.
The network worked before there were ever non-mining nodes having a copy.
So if only the miners have a copy of the transactions then it is more than good enough.Your example is like a straw poll. The non-mining nodes had no real authority to actually enforce their conclusion.
Like I said the only vote that holds authority is if you win a block, then your vote on the previous block validity counts as a confirmation.
If you want to be PayPal, it is good enough for only miners to have a copy. All nodes enforce the protocol regardless of your insistence that they do not. Look at the code.