No, you have it backwards: It would be just as if the miners who are mining invalid blocks did not exist.
Any miner building further upon an invalid block would mining on invalid chain, and thus similarly would be ignored.
Bitcoin is decentralized. This means that no one has the authority to declare a block invalid. The only person who has this authority is drawn at random according to their demonstration of Proof of Work.
Make sense now?
If nobody can declare a block invalid, then why can't I mine a block that pays myself 1,000,000 bitcoins?