Think of bitcoin as virtual cash instead of bank deposits. As you keep your cash in a wallet you keep bitcoins on storage media like your PC hard disk or an external hard disk/memory stick.

One advantage of bitcoins: since it is virtual money you can copy it to multiple locations for safety. The bitcoin concept ensures that you can have many copies of the same bitcoin units (technically not 100% exact though) but you can only spend them once. Your wallet could burn in a fire (if you left it at home).. If you store your bitcoins on multiple distributed HDDs redundantly the chance of loosing them is much smaller. You could also print them on paper and keep that in a safe.

For security you need to ensure that nobody including malware gains access to the bitcoins. I.e. my laptop HDD and external HDDs are encrypted using TrueCrypt and my Bitcoin wallet is encrypted itself.

You can also store your bitcoins only as Endgame says. The handling might be easier -- it is much like bringing your cash to a bank though: you have to trust the online wallet or cloud storage provider. Only difference: as far as I know bitcoin online wallets don't create virtual credit money yet and hope that not all depositors come to get their bitcoins back (bank run). Although nobody can prevent bitcoin online wallets/"banks" from doing the same thing..