Anyway, is it possible to create separate "specific-use-only" wallets you could store in, say, your phone or your Trezor, where funds could only be sent to specific whitelisted addresses?
I don't see a reason to create such a thing, there already exist gift cards (as a private payment system)
The idea is that the coins in the wallet could only be sent to specific addresses -- legitimate merchants. If a thief demanded your bitcoins, he'd have to steal the entire physical wallet device and could only spend the coins at legitimate merchants. He could not simply transfer coins to his own wallet. If the hardware wallet were stolen, the police can easily put together a database of blacklisted addresses which are pushed to merchants (this could be very effective if bitcoin change could be forced to go into old addresses instead of generating new ones). This DOES NOT affect fungibility.
I image a hardware wallet as a device is a device that can be purchased anonymously. It would be cheap, and designed to cheaply connect to a public network wirelessly to send transactions and monitor the blockchain. You would use it to transfer money to another private person's wallet or the pay terminal of a retailer. It's not designed to hold a lot of money, that is, more money than you are willing to lose. It would be analogous to filling your own wallet with cash for the cash purchases you intend to make for the day. The reason people may want anonymous wallets is the same reason today you would choose to buy something with cash rather than using a credit card. You may not want the receiver to be able to know who you are.
Now, if you lose your hardware wallet, you can go home and sweep the addresses that it owns with the optional wallet backup program. Same thing if a thief physically takes your wallet, you can sweep the wallet with the backup program before the thief does, you get your money back. If the thief sweeps the wallet before you do (after cracking or coercing the password on the device such as a 4 digit pin), then the police can track the money until it hits a public address that may unblind the thief (i.e. a retail pay terminal with surveillance.)
If you keep a large amount of money in such a wallet, then you may be asking for trouble. There's always a small possibility that manufacturer of the wallet may have inserted or allowed a hack that allows someone else to gain access to the wallet's private keys.
The idea of a hardware wallet is that it would become an acceptable risk to carry certain amounts of money, and the device or the money can be lost without causing a catastrophic loss of funds. There's no reason to implement a specialized limited fungibility system for bitcoin, the retailer can just sell you a gift card instead.