The way you put it now basically says 'Your wallet can only be as secure as the hardware & software it is directly running on'.
That's not even 100% correct, since you could have an infected device, but it's using heavy sandboxing and the virus can't reach the 'wallet sandbox' or something like that.
Anti-sandbox and Anti-VM types of malware and password stealers have been around for years. I remember seeing them back in the days of pirated and warez software. Even then it was possible for certain malware to detect that a system is using a sandbox or virtual machine and break through its defenses to perform any kind of attack it was designed to perform. I wouldn't rely on a sandbox as an ultimate way of protection. Being careful and not opening and executing programs and scripts on your end is still the best protection.