Not necessarily. For hardware wallets, the attacker still needs to extract the seed. On Trezor devices we now know that can be done for less than $100 with fairly basic components. As far as we know, other devices still require high tech equipment and the expertise to use it. Very few people have the access and skills required, and hiring such a lab would cost several thousand dollars at a minimum. It's simply not a concern for your average crypto user, but is a concern for people holding large sums of money who are being specifically targeted by criminals.

Nothing is 100% safe, and if you have access to additional options (such as passphrases) which increase your security, then I would always opt to use them.