You can have a 3rd party verify all the key math, other than the concealed key in the coin. You have to trust the right key is in the coin. I have good control over this with multiple checks and balances.
This is what I love about your product, Casascius. Basically, we trust the "inherent" value of a dollar bill because it's backed by the US government, and we even somewhat trust the "inherent" value of bitcoins based on their usefulness and speculative demand, but you've managed to redirect the only support of value in your physical bitcoins to...you.
I am sure you're ok with your product just being a novelty, though.
The cool thing is though, that I don't have a monopoly on this. Decentralization doesn't necessarily mean "never trust anybody". You trust your grocery store not to sell you expired/spoiled/poisonous food; if they did, you would have somewhere else to shop. Surely, you wouldn't say your local grocery store has cornered the market on food, otherwise I would expect that you only eat things you grow yourself.
I am eager to see someone else do a great (or better) job of Physical Bitcoins, meanwhile, I have just laser-cut 1,700 more private key circles this afternoon, all with "1Agxxxxx" addresses, meant for my silver coins.