Blockchain is a web wallet, and you might also call it your client.
Or more specifically a web based hybrid wallet.
Pure web wallets (such as coinbase, localbitcoins, and bitstamp) do not provide you exclusive access to your private keys.
Hybrid wallets (as long as they haven't been maliciously modified) don't have access to your private keys in decrypted form, don't have access to your passphrase, and provide you with a backup of the wallet so you can access it even if the website disappears.
Address = Thing people need to send bitcoins to you,
Private Key = Thing you need to spend bitcoins (Keep this secret)
More specifically, the private key is the thing that the wallet software needs to spend the bitcoins. As long as you have a good backup of the file that the wallet uses to store the private keys, and you know the passphrase that is used to decrypt the contents of the file, you don't ever need to actually know what the private keys are. Furthermore, even if you do know what the private keys are, you'll find it VERY DIFFICULT to create and broadcast a transaction without providing those private keys to some sort of wallet software that you'll use to generate the transaction.
And you can export the private key. The public key is technically just the address.
Technically, the address is a hash of the public key. Even more specifically, it is a hash of a hash with a version and a checksum.