Even if an attacker is able to distinguish a USB flash drive from a hardware wallet, it will not be easy to use it. Let's say the attacker took possession of my hardware wallet (Ledger Nano S), which has an eight-digit password. In the case of three wrong combinations, the hardware wallet resets all settings to the initial state and the attacker simply can not get my coins.
Device will reset if wrong PIN is entered 3 times in a row, but smart hacker will not try to obtain your PIN in that way. They will try to hack it with brute force, and 8 digit PIN is very limited in number of combination. I'm not sure what kind of equipment is needed and whether Ledger has some protection to prevent such hacking attempts (in case your wallet is stolen).
But let's say a PIN of 8-10 digits is small joke for any supercomputer or botnet :
To demonstrate the importance of password complexity, let's start with a pincode password such as "123456789". In this case, the character set (0123456789) consists of 10 characters. For a 9 digit password using this character set, there are 10^9 possible password combinations. Therefore, it will take (1.7*10^-6 * 10^9) seconds / 2, or 14.17 minutes, to break this password on average. On a supercomputer or botnet, we divide this by 100000, so it would take 0.0085 seconds to break a password.
Because of facts above, using of passphrase on hardware wallet is very desirable. Of course, only if the user knows what he is doing.
https://support.ledger.com/hc/en-us/articles/115005214529-Advanced-passphrase-security