Not if you use hardware wallet in a clever way... haha
You have said it all, though very short but it is a very good statement. Hardwallet is the best wallet to be used to store our coin(s), it has been trusted and tested over time with better reliability than any other way of storing the coins. As you said, we must be clever about it, and this goes a whole long way even as the word clever is so huge but the holder of the Hardwallets is often taken for granted. Many believed that they had their wallet in a hard wallet but were never clever about it but they had their coins still stolen all the same.
This calls for smartness in handling it. Many stories about this are online in this regard and the same are advised on the same online to prevent such. It can't hurt for anyone having such wallets to seek such knowledge so that they can effectively protect their assets. This is especially true as the person who could deem it fit to buy a hardwallet to hold their coin must have a sizable value of the coin(s) in there to warrant such buying. "What worth doing is worth doing well."