Apparently, there is a vector attack where the address in the Wallet UI is the user's one, and the one on the HW wallet is the attacker's one. So use extreme caution.
Wait.... are you saying there are people who use a hardware wallet but don't confirm the address on the screen? That's one of the main reasons to use a hardware wallet: to verify you actually sign a transaction to the correct address!
I'd be much more concerned if an address changes when I read it on my screen. Let's say you deposit to an exchange, but the deposit address on your screen is changed already before you copy it. That would mean the address on the hardware wallet matches the address on your monitor, but it's now the exchange's address.