How could that have happened?

They changed the code that calculates the block size without understanding how it works, and pushed out the change without any review or testing. This is standard practice for Bitcoin Unlimited developers, and nobody is the least bit surprised that it blew up in their faces. It may be the first time Bitcoin Unlimited users have lost money as a direct result of the developers' incompetence, but it's unlikely to be the last.
leaving the connection open would have done no harm because core nodes already rejected the block..
It harms nodes with limited connection slots. Banning misbehaving nodes frees up slots for correctly behaving nodes. Any node that relays an invalid block is either defective or malicious, and there's no point in maintaining a connection to such nodes.
but by dropping the connections then no longer plays by consensus rules and the two non-communicating nodes could end up following different data. because they no longer allow communication with each other to reject/accept the same data.
They're already following different data, whether they're communicating or not. By definition, there can be no consensus when one node thinks a block is valid and another does not.