Ok, it looks like I finally have this fixed.  It was just the orphan blocks which were throwing me off, and now those are all taken into account.  The parser now successfully reads every single block, transaction, input, and output in the blockchain.  There are some output scripts which it fails to derive a valid public key signature for  However, in all of those cases I found both blockexplorer and blockchain.info have the same issue, so I'm not treating that is a big concern for now.

I plan to write up a specific blog post detailing all of these 'gotchas' you have to know about to stream/parse the block-chain sequentially (rather than trying to scan all of the blocks into some kind of graph/tree) up front.  There are a bunch little annoying things that are not obvious.

[Edit: Spoke too soon, I get off the rail a little further down the blockchain, still have to debug this some more...]

John