As of today, I think the chances of getting a computer virus that infect secure-element-less hardware wallets through USB and then extract your seed, are very slim. The main benefit of such a storage chip is still against physical attacks, since it's obviously trivial to read off of a memory chip on a PCB in- or out-of-circuit.
One pretty big mitigation against this is using an airgapped hardware wallet instead.

Of course, if you input your xpub into a wallet application on a compromised 'China-made blackbox', the xpub could be stolen through some CPU backdoor.

It's not true though that 'no one knows what it [secure element chip] does'. I noticed a few years ago, more sophisticated secure chips were more 'en vogue'; getting not only the task of secure storage but also being used for seed generation and such. These days, e.g. Foundation Passport, uses a more simple chip that just stores data and nothing else. The RNG is completely open-source on the PCB.

But I digress; for life-changing amounts of money, always feel free to go more paranoid (I don't intend to discredit / discourage when using this word), roll dice, use minimal, open-source software that runs on a RISC-V CPU, and store everything on paper!