Quote from: salfter on April 17, 2014, 11:45:45 PM

I've let the current version grind away for a week or two, only to get stuck on importing the Bitcoin blockchain. On block 234062, it fails pretty spectacularly. I probably shouldn't have been running this branch anyway as it doesn't handle scrypt coins, but that still doesn't explain what's going horribly wrong. I'm running bitcoind 0.9.1 and told it to verify the entire blockchain; it came back with nothing wrong. All this is on a freshly setup Ubuntu Server LTS 14.04 box (had been running late betas or release candidates); I'm using PostgreSQL for the database.

What follows is what Abe.py spits out when I try to have it resume. Take particular note of the line that starts with "InvalidOperation: Invalid literal for Decimal:"

Code:

Opened /home/bitcoind/.bitcoin/blocks/blk00057.dat
Exception at 74222644
Failed to catch up {'blkfile_offset': 73974221, 'blkfile_number': 100057, 'chain_id': 1, 'loader': u'blkfile', 'conf': None, 'dirname': '/home/bitcoind/.bitcoin', 'id': Decimal('1')}
Traceback (most recent call last):
File "/home/bitcoin-abe/bitcoin-abe/Abe/DataStore.py", line 2503, in catch_up
store.catch_up_dir(dircfg)
File "/home/bitcoin-abe/bitcoin-abe/Abe/DataStore.py", line 2767, in catch_up_dir
store.import_blkdat(dircfg, ds, blkfile['name'])
File "/home/bitcoin-abe/bitcoin-abe/Abe/DataStore.py", line 2899, in import_blkdat
store.import_block(b, chain = chain)
File "/home/bitcoin-abe/bitcoin-abe/Abe/DataStore.py", line 1043, in import_block
tx['tx_id'] = store.import_and_commit_tx(tx, pos == 0, chain)
File "/home/bitcoin-abe/bitcoin-abe/Abe/DataStore.py", line 1852, in import_and_commit_tx
tx_id = store.import_tx(tx, is_coinbase, chain)
File "/home/bitcoin-abe/bitcoin-abe/Abe/DataStore.py", line 1819, in import_tx
txin['prevout_hash'], txin['prevout_n'])
File "/home/bitcoin-abe/bitcoin-abe/Abe/DataStore.py", line 2431, in lookup_txout
(store.hashin(tx_hash), txout_pos))
File "/home/bitcoin-abe/bitcoin-abe/Abe/DataStore.py", line 251, in selectrow
return store._sql.selectrow(stmt, params)
File "/home/bitcoin-abe/bitcoin-abe/Abe/SqlAbstraction.py", line 479, in selectrow
ret = sql.cursor().fetchone()
File "/usr/lib/python2.7/decimal.py", line 548, in __new__
"Invalid literal for Decimal: %r" % value)
File "/usr/lib/python2.7/decimal.py", line 3872, in _raise_error
raise error(explanation)
InvalidOperation: Invalid literal for Decimal: 'N26...'

Yeah wow, it's a doozy! As near as I can make out, lookup_txout is getting that 17kB string as its txout_pos argument, so import_tx must have got it in txin['prevout_n'], which it got from tx. Since you are loading from blkfile, tx must have come from chain.ds_parse_block (DataStore.py Line 2889) and ultimately from deserialize.parse_Transaction. That gets txin from parse_TxIn, which gets txin['prevout_n'] from BCDataStream's read_uint32 method, and ultimately from the standard struct.unpack_from function. This whole code path is very common in Abe and must have run millions of times successfully during your load before the error.

I would really like to know whether struct.unpack_from returns a 17kB string and, if not, where my chain of reasoning fails. If so, it seems like a serious Python bug, and I would like to create a minimal test case and report it.

Scrypt coin configuration could be patched in after the fact, so you are okay there.