I've only used block chain. Info and coin base. Is block chain my client?
Yes. You can choose to use additional wallets if you want. I generally advise using 3 wallets if you have an amount of bitcoins that you feel is significant.
What is the difference between address and keys?
A private key is used by the wallet software to sign transactions to prove that you have the right to spend the bitcoins.
A public key is calculated from the private key. This is a one-way calculation. There is no known feasible way to calculate a private key if you only know the public key. The public key is used to verify that the signature on a transaction is valid.
A bitcoin address is a hash of a public key. This is another one-way calculation. There is no known feasible way to calculate a public key if you only know the bitcoin address. Using a bitcoin address reduces the amount of information that you have to give someone to send you bitcoins (a 160 bit hash instead of a 256 bit public key). It also adds an additional layer of protection since there are multiple one-way calculations between the private key and the bitcoin address.
What client should I use so I always have my bitcoins and the client is only doing the transaction nothing else?
You never have your bitcoins. Bitcoins don't actually exist. The term "bitcoins" is an abstraction that we humans use to make it easier to discuss the concept of the transfer of value.
What actually exists is a transaction ledger called a "blockchain". This transaction ledger encumbers an amount of value with a requirement that a signature from a particular private key be provided to re-encumber that value with a new requirement.
When you "send bitcoins", what you actually do is provide the digital signature that proves that you have the right to re-encumber some amount of value. Then you encumber that value with a new requirement of a digital signature that someone (the recipient) else is able to provide.
To make sure that you can transfer value from your control to someone else's control, you have to make sure that you have all the necessary private keys to generate the proper signatures. Your wallet software uses these private keys to generate the signature for you. Most wallet software stores these private keys for you, but you should have a good backup just in case something happens to the copy that the wallet is using.
Do I get my private key and public key from this client?
Since software handles the storage, and security of the private keys, it is generally not necessary for an individual to know what their private keys are. If you actually want to know your private keys, then yes you'll need to get them from the wallet.
It is never necessary to know the public key.
You'll also get the bitcoin addresses that are associated with the private keys from the wallet.