There has been no demonstrated successful physical attack against Ledger products, but that is not to say one doesn't exist. With an electron microscope and enough time and expertise, then it is like that even the secure element will be crackable and the seed able to be extracted, but we are now probably talking about in the order of weeks at a cost of several hundred thousand dollars. This differs obviously from Trezor wallets which can have the seed extracted for less than a hundred dollars in the space of a few minutes. Any hardware wallet shouldn't be viewed as infallible, but rather as a mechanism to buy you (hopefully plenty of) time to move your coins to new addresses.