You actually read these stories from time to time. However, I consider it almost impossible that the private key could be extracted by an actual attack without access to the physical device. In this context, it is much more common to read that users have downloaded phishing software such as a fake Ledger Live and were asked to enter their mnemonic code:
Ledger live was compromised for me
Even the best hardware wallet won't help if the user falls for such attacks.

In principle, however, I agree with you: you have to be damn careful what you do and should never be lulled into a false sense of security, as this is guaranteed to backfire.