I don't think the concern is over how many coins you are storing on the hardware wallet, but rather, making yourself a target for attackers. Attackers who see a hardware wallet (and know what it is) don't know how many coins you are or are not storing on it, and most will assume you are using it for larger sums which would be unwise to keep on a mobile wallet. Indeed, using a hardware wallet for trivial amounts such as in the region of $50 might be more of an issue, since if you are attacked, a thief is unlikely to believe that you only have $50 worth of coins on the device and will continue to push you for more which you will not be able to provide.

I do use hardware wallets when I travel, but I don't use them in public and instead would use them to top up the mobile wallet on my phone as and when necessary, which is the wallet I will spend from in public.