Whether you choose to have that privatekey online with a 3rd party; in your computer; on your phone; on paper; or in your brain; or even in your brain as something that can be translated back into the privatekey, that is all it is. The public key is then trivial to derive from the private key. The balance then is simply the sum of all transactions.
Okay now thinking through this, I am getting to understand this. So basically one can generate the bitcoin address (from any of the multiple options online). And the balance as you say is "the sum of transactions" sent to that address. And I assume I would need an app or some other way to use the private key when I want to spend the BTC.
One question -- Is there a way to check the balance? Would it be as easy as peasy as just looking at:
http://blockexplorer.com/address/?