The Trezor seems to be a popular hardware storage method. A printed private key in a lockbox isn't a bad idea either. Some people even keep their BTC on physical coins. Casascius coins being arguably the most popular.
Websites have a bad track record of getting hacked at the end user's expense. Most will charge you a fee to withdraw funds, but it's worth it.
Thanks for that info. The possibility of storing currency in a physical coin is very fascinating.
On the Casascius coins, they ARE fascinating. And very collectible. Nearly all of them come loaded with Bitcoin and most people don't add any more Bitcoin to them, it's generally seen as degrading the collectibility of the coin. That said, there are other coins by other makers that come as blanks and can be used as a physical wallet, the user loads the bitcoin on the coin itself. Check out the Collectibles board. And The Bitcoin Penny company comes with the most affordable versions of do-it-your self physical coins that can be loaded with Bitcoin.
On the risk of an online "website" wallet getting hacked. It's less than OG Nasty implies. Their track record is significantly more supportive of conducting their business fairly and safely.
There was a major hack to MtGox a few years back and there have been a few outages and isolated incidents. But there are several reputable online wallets that have never been hacked. The greatest issue that comes with these online wallets is local governments creating greater restrictions over how these companies operate, which leads the online wallet company to choose to discontinue their service to people who live in specific locations. E.g., Coinbase recently discontinued offering service to people living in Hawaii or Wyoming in the US.
The broader concern about using online wallets (which manifested itself in OG Nasty's "getting hacked" comment) is that these companies don't allow you, the account holder, to own the private keys of your wallet. This means that the online wallet company has the real control over your Bitcoin and you have to trust that they won't f*ck you over.
The same applies to fiat banks. When you deposit your paycheck in your local bank your money doesn't sit in a safe deposit box with your name on it waiting for you to make a withdrawal. Your money goes into the banks' general pool of funds to be used to loan out, pay other account owner withdrawals, pay their bills, etc. You're left with an "IOU" that the bank will pay you back. The only difference is that there is greater regulation over fiat banks, and therefore you can sue them more easily if they take off with your money.
Proceed with caution...at least now you're more educated!