A good hardware wallet is more than just a simple USB drive. The whole point of hardware wallets is exactly that you can plug them in to compromised or online computers without having your private keys stolen.

The most that a compromised computer could do is craft a malicious transaction and push it to your hardware wallet to be signed. As long as you are paying attention and you don't confirm the malicious transaction on your hardware wallet, then your coins remain safe.