Blockchain.info is cool though, because the site owner doesn't have access to the private keys of the users.
That is not entirely true.
The blockchain.info wallet is code that runs in the browsers, if it can work with your bitcoins after you've unlocked your wallet, and they provide the code, then it stands to reason that they can modify the code to take actions using your private key after you have unlocked it.
Or simply, they can change their javascript to be malicious. It is still a trust based service.