Just because you use your PC to communicate with your hardware wallet, the keys are not handled by the PC. The crucial system here is the hardware wallet. I could have made that clearer.
A hardware wallet can be only as secure as the hardware is (e.g. vulnerabilities in the MCU or SE).
The same applies to a software wallet, taking hardware, software, network connectivity etc. into account.
Oh right, sorry, I thought you were trying to say 'Your wallet can only be as secure as the
system it is running on [== the OS / computer used with it] / the building it is placed in.'
The cool thing about hardware wallets is that they're not powered on neither connected to networks constantly; thus even if we assumed the same 'system security' as your daily driver laptop, the attack success probability would be lower since the time windows during which attacks are possible, are shorter. Also these devices don't run third party software, unlike Windows PCs where many programs run as root ('admin') or other desktop OSes where 3rd party software simply runs on them directly (privileged or not); thus reducing the overall 'underlying system security' as well.