The way I try to explain it to less technically savvy people is like this:

Imagine a gigantic wall made of hollow glass bricks. They're clear on your side and opaque on the reverse side. Each brick has a unique number identifier on it. And each brick has money inside it or perhaps no money at all. Are you imagining this big wall? Good. Now imagine that anyone in the world can walk up to any brick they like and see how much money is inside. Since this is usuable by the whole world, you cannot use ordinary US dollars, or Chinese RenMinBi, or Russian rubles. You would first have to buy special tokens. These tokens all have the same value but each world currency would buy a different amount of tokens. There is even a slot where anyone can put any amount of tokens they like into any brick they like without the knowledge or consent of the owner of that brick. There is no way of knowing who owns which brick(s) and there is no way to get the tokens out from this side of the wall. Attached to each brick is a log that shows each time tokens were put in and taken out and records which bricks the tokens came from and where they went to, but not the owners of those bricks, which, as I said before, no one actually knows.

On the reverse side of the wall, all of the bricks are opaque but there is a keyhole in each which opens with the correct key. If you had the key and knew which brick to open you could remove any amount of tokens from it. It would not matter who rightfully "owns" it. It only matters if you have the key. (Although you could try any random key on as many keyholes as you like, the possibility of it actually working is so minute that it essentially is impossible. You would have better odds of picking the winning lottery numbers on the next 1,000 drawings in a row!) Multiple people, however, could have the key if the "owner" either gave it to them or was careless with his and allowed them to copy it. I say the wall is opaque from the other side because there is no way of seeing who is removing tokens from each brick, you can only see where the tokens move to and from. If you "owned" a brick, you could give your public address to someone and they could deposit tokens into it but they would have no way of getting those tokens back without the private key to open that brick and remove them.

Think of the blockchain this way. It exists "out there" and anyone in the world can see it and can see the balances of each address. But no one knows exactly who owns which address. The public key is the address of that brick. The private key is the key that opens that brick. And "bitcoin" is simply the unit of measurement or the name we give to the numbers of tokens that make up each account. (I know this is not 100% accurate but for sake of simplicity I explain this way.) In the real world you need three things to own and spend Bitcoin: a public key, a private key, and some sort of software connected to the Internet that knows how to interact with the Blockchain. This is what your wallet is. It is the software that "talks" to the Blockchain. You can give it one or any number of public/private pairs you like and it will take care of the rest. It is not necessary to back up a wallet. The only thing you actually have to back up are the keys, which are just a unique series of letters and numbers or, in reality, two unfathomably huge numbers. There is no limit to how many addresses you could own. If I ever sent or received Bitcoin from "you" then I could remember that you owned that address and can now see all of your transaction history if I like. With some effort, I could trace every bitcoin you ever sent and where it was sent to. As a side note, work is currently being done on "mixers" which scramble this information preventing any way of tracing transactions back to their source.

You can always change, remove, or install a new wallet without worrying about transferring balances from one wallet to the other. The only things you would have to transfer are the keys. The balance does not exist on your computer nor in the wallet. It exists only as the sum of all transactions attached to the public address on the Blockchain, this nebulous ledger which exists "out there" and is maintained by the collective agreement of a myriad of computers and math. The Bitcoin protocol and the Blockchain seem to be, at least so far, unbreakable. The real danger in Bitcoin is currently in the wallet. If someone had access to your computer or smart phone, they could potentially have access to your wallet and there would be no recourse and no one to complain to if your account was cleaned out. This is why I go to such great lengths to explain the basics behind the Blockchain. Because ultimately, security is up to the user, not the software. A Bitcoin wallet can be the most secure software ever created but if you keep your public/private pair written on a piece of paper in your leather wallet in your pocket, then you would be at risk.