No, you have it backwards: It would be just as if the miners who are mining invalid blocks did not exist.
Any miner building further upon an invalid block would mining on invalid chain, and thus similarly would be ignored.
Bitcoin is decentralized. This means that no one has the authority to declare a block invalid. The only person who has this authority is drawn at random according to their demonstration of Proof of Work.
Make sense now?
Bitcoin is decentralized. This means that no one has the authority to force full nodes to accept invalid blocks. Miners have no such authority, drawn at random or otherwise.