You're missing the simple fact that Hardware Wallets, while some of them might have the appearance of one, are not just a plain USB stick. They have custom hardware/software/firmware that prevents external devices from accessing the data stored within in.

Essentially they provide a limited "API" that external devices/software have to use to communicate with the device, such that the sensitive information stored within it (ie. seed/private keys) cannot be extracted using that API... at least, in theory