Your real wallet is in the device. The PC on your desk can keep a copy of the public keys so that it can show you your current balance as a convenience, but that's it. When you want to pay, it sends the destination address and the amount, only. The device then creates the transaction all by itself, using transaction records it already knows.
Yes, the private keys are stored only on the secure device, but that wasn't my question. My question was, if the client can create the required unsigned transaction, which requires nothing more than knowledge of the public block chain, why does the secure device need any more data than just that unsigned transaction, which it then signs. Everything but the actual transaction signature is public knowledge, the only thing the device needs is the transaction itself, so that it can properly sign it. It does not need to know all other unrelated transactions that I've done in the past, and it has no way of knowing whether to trust them anyway.