This is one of the advantages to having a multi-sig wallet where you need multiple keys to sign a transaction instead of just one. Having a single key represents a single point of failure. Of course you still need to manage your keys in a multi-sig setup, but you don't risk ruin if you happen to lose one of the keys (if for example you had a 2 of 3 setup which means you have 3 keys but need only 2 to sign a transaction).

Lots of options to think about in terms of security. A bank deposit box, safe, home of a close relative, multiple physical locations, etc. Think about the tradeoffs between security and accessibility (more secure may limit accessibility, more accessibility may limit security).