I think you're referring to
Dark-Skippy. ... In this case, the chances of it going unnoticed by the user, even an advanced and veteran user, are very high.
No, I didn't have this malicious signer stuff in mind. I was referring to malware that could interfere with the software wallet portion that talks to your non-infected hardware wallet. Malware could patch the software wallet and intercept at points when transaction data is prepared to be handed over to the hardware wallet signing device. The hardware wallet signing device would receive an already altered transaction with manipulated outputs that moves coins in favor of the attacker.
It would only be successful if the user doesn't control carefully all details of the transaction outputs (better check the inputs, too).
But true, Dark-Skippy is pretty evil stuff and hardly noticeable if and how it leaks crucial data from your most precious wallet's secrets.