Mining pools started confirming these transactions, instead of blindly following Bitcoin Core.

If you have a block template, that you can get through "getblocktemplate", then it contains transactions, that are flying in different mempools. As long as this template is identical in most nodes, you have all transactions already validated, when the new block is mined. Then, verifying a given block is very fast.

However, if some mining pool produces a block, which contains transactions, that were never broadcasted before, then nodes have to download these transactions, and verify them, to know, if a block is valid or not.

And this is the main reason, why developers decided to lift more limits. Because they are already lifted by mining pools. So, keeping them does not change anything, and only makes everything slightly slower, if some node actively rejects some transactions, and they are later downloaded again, when a new block is produced.

Which also means, that there is a new attack, that can be done, and that cannot be easily stopped: miners could decide to produce 4 MB blocks, with self-transfers, which would be costly to validate, and which would break any standardness rules, while also being valid, when it comes to consensus rules. And then, if miners will keep breaking next limits, then developers will keep lifting them. And then, the end result would be a chain, where only consensus rules are enforced, and there is no longer any concept of "non-standard transaction", so there is no obvious way to make new soft-forks, without burning someone's coins.