Yea, but this is a physical attack and still need the right tools and skills to take advantage of it, so the likelihood of such an attack being taken place is close to 0, though if someone has access to the device, knowledgeable and know it would be worth it (and assuming that there isnt additional security measures in place), they could do it. I do agree that people should be using a additional passphrase with their hw wallet (or any wallet in that matter).

However, with ledger, with it being close source, we dont know of any vulnerabilities that may not be fixable (I suspect that their ledger blue has an security bug that cannot be fixed, though it could just be that they know that was really a failure product). Furthermore, with them relying on a third party security chip (secure element), that raises alot more question about how reliable such a third party may be, if there are unknown backdoors, etc.