The only different between Normal USB (holds the key) and Hardware wallets, that hardware wallets have extras such as Connect to Wallet by Wi-Fi or Bluetooth, and Built-in software which communicates with your wallet.
Hardware wallets are very different from your USB. Hardware wallets contains the necessary components (MCUs) which is used to generate the signature using the private keys stored on it. LiveUSBs cannot do so and it relies on the computer for the signature which makes it vulnerable to BIOS root kits as well though that would be more complicated.
If you're talking about something like an OpenDime, it generates the key on the device itself and gives you the address. You will reveal the private key only if you poke the seal, pretty cool!
I can call that an extra software. Hardware wallet is a device holding private keys with extra security & hardwares (ex. Bluetooth) & software. I was just have the same OP's question perviously. I was seeing ledger nano ads and confused how bitcoins are stored on such a usb. Now i see its just a key file stored into it, and all rest are just extras.