You can have a 3rd party verify all the key math, other than the concealed key in the coin. You have to trust the right key is in the coin. I have good control over this with multiple checks and balances.
This is what I love about your product, Casascius. Basically, we trust the "inherent" value of a dollar bill because it's backed by the US government, and we even somewhat trust the "inherent" value of bitcoins based on their usefulness and speculative demand, but you've managed to redirect the only support of value in your physical bitcoins to...you.
I am sure you're ok with your product just being a novelty, though.