Yea, but this is a physical attack and still need the right tools and skills to take advantage of it, so the likelihood of such an attack being taken place is close to 0, though if someone has access to the device, knowledgeable and know it would be worth it (and assuming that there isnt additional security measures in place), they could do it. I do agree that people should be using a additional passphrase with their hw wallet (or any wallet in that matter).

However, with ledger, with it being close source, we dont know of any vulnerabilities that may not be fixable (I suspect that their ledger blue has an security bug that cannot be fixed, though it could just be that they know that was really a failure product). Furthermore, with them relying on a third party security chip (secure element), that raises alot more question about how reliable such a third party may be, if there are unknown backdoors, etc.

https://www.reddit.com/r/ledgerwallet/comments/6vgl1z/is_the_nano_ss_firmware_open_source/
btchip Ledger Innovation Lead & Co-Founder 2 years ago
The applications are Open Source and available on https://github.com/LedgerHQ

The firmware itself is not Open Source yet, but most parts will be in the future (see https://blog.ledger.co/secure-hardware-and-open-source-ecd26579d839 for an architecture description). In the meantime a motivated party can verify that the isolation works as described.

https://www.reddit.com/r/ledgerwallet/comments/amsc3t/is_ledger_open_source/
btchip Ledger Innovation Lead & Co-Founder 10 months ago
The applications are Open Source, more and more parts of the Operating System itself will be opened over time. We've chosen this architecture because it's not possible to achieve the level of physical security we're looking for with generic chips today. For more information about our architecture you can check https://www.ledger.fr/2016/06/09/secure-hardware-and-open-source/