That's right: if you have one piece of software that everybody is running, you can hit many more users when you find a bug in it.
I believe this is the main counter-argument against my idea of a HW wallet firmware, but on the other hand, we're all running the exact same Bitcoin Core v22.0....
True, but Bitcoin Core isn't the only open-source Bitcoin software wallet. It is the main full-node client, yes. But if a serious bug gets discovered in Core, the same vulnerability wouldn't be present in Electrum, for example. Or any other client that doesn't use that exact code. If we had one widely accepted code or a piece of it replicated across multiple clients, we would have exactly that. A longer list of possible targets and victims.