Yes, my concern is primarily key logging.
I think your concern is well founded. many have come here to announce that they have been "hacked" via a key logger. The good news is that there are many things you can do to offset the risk. Besides anti-malware software and avoiding installing anything suspicious, it is important to keep any large-ish balance offline. You can create a wallet offline and then print out the private key. Then delete the wallet and your good. The attacker has no path to your coins.