Quote from: Gavin Andresen on April 05, 2012, 05:58:32 PM

Please file bug reports on the github issue tracker. Include enough information so we can reproduce the problem (what platform? what seems to cause the problem? etc) and it might get fixed.

Another crash happened today, same platform etc.

This time it ran with -printtoconsole (no -debug). The scrollback buffer is filled with hundreds of

Code:

received getdata for: tx 7680e65403ab1b258887

.. then:

Code:

received getdata for: tx 7680e65403ab1b258887
received getdata for: tx 7680e65403ab1b258887
trying connection 81.176.229.178:8333 lastseen=-2.5hrs
connect() failed after select(): Connection refused
received getdata for: tx 7680e65403ab1b258887
received getdata for: tx 7680e65403ab1b258887
received getdata for: tx 7680e65403ab1b258887
received getdata for: tx 7680e65403ab1b258887
received getdata for: tx 7680e65403ab1b258887
received getdata for: tx 7680e65403ab1b258887
askfor tx 263d4ec0b5712e6e1b47 0
sending getdata: tx 263d4ec0b5712e6e1b47
trying connection 141.106.36.16:8333 lastseen=-3.9hrs
connect() failed after select(): Connection refused
received getdata for: tx 7680e65403ab1b258887
askfor tx 263d4ec0b5712e6e1b47 1333717043000000
askfor tx 263d4ec0b5712e6e1b47 1333717163000000
askfor tx 263d4ec0b5712e6e1b47 1333717283000000
askfor tx 263d4ec0b5712e6e1b47 1333717403000000
askfor tx 263d4ec0b5712e6e1b47 1333717523000000
askfor tx 263d4ec0b5712e6e1b47 1333717643000000
askfor tx 263d4ec0b5712e6e1b47 1333717763000000
askfor tx 263d4ec0b5712e6e1b47 1333717883000000
askfor tx 263d4ec0b5712e6e1b47 1333718003000000
askfor tx 263d4ec0b5712e6e1b47 1333718123000000
AcceptToMemoryPoolUnchecked(): size 22
AcceptToMemoryPool(): accepted 263d4ec0b5
received getdata for: tx 7680e65403ab1b258887
received getdata for: tx 7680e65403ab1b258887
trying connection 69.119.102.53:8333 lastseen=-2.9hrs
sending getdata: tx 35f44d3defc623e850d4
received getdata for: tx 89c54bace5d7221fb8e2
ERROR: AcceptToMemoryPool() : not enough fees
received getdata for: tx 7680e65403ab1b258887
Added 1 addresses from 74.58.231.130: 3415 tried, 8183 new
Added 1 addresses from 95.211.10.8: 3415 tried, 8183 new
received getdata for: tx 7680e65403ab1b258887
askfor tx fee32af70d821d823ba6 0
sending getdata: tx fee32af70d821d823ba6
askfor tx fee32af70d821d823ba6 1333717046000000
askfor tx fee32af70d821d823ba6 1333717166000000
askfor tx fee32af70d821d823ba6 1333717286000000
askfor tx fee32af70d821d823ba6 1333717406000000
askfor tx fee32af70d821d823ba6 1333717526000000
AcceptToMemoryPoolUnchecked(): size 23
AcceptToMemoryPool(): accepted fee32af70d
received getdata for: tx fee32af70d821d823ba6
Added 1 addresses from 91.95.240.5: 3415 tried, 8183 new
Added 1 addresses from 68.38.31.2: 3415 tried, 8183 new
askfor tx a817688a1f0463c40287 0
sending getdata: tx a817688a1f0463c40287
askfor tx a817688a1f0463c40287 1333717048000000
askfor tx a817688a1f0463c40287 1333717168000000
connection timeout
askfor tx a817688a1f0463c40287 1333717288000000
accepted connection 147.133.197.3:64460
askfor tx a817688a1f0463c40287 1333717408000000
askfor tx a817688a1f0463c40287 1333717528000000
askfor tx a817688a1f0463c40287 1333717648000000
AcceptToMemoryPoolUnchecked(): size 24
AcceptToMemoryPool(): accepted a817688a1f
version message: version 32300, blocks=174516
trying connection 83.101.77.156:8333 lastseen=-3.7hrs
connect() failed after select(): Connection refused
received getdata for: tx a817688a1f0463c40287
received getdata for: tx a817688a1f0463c40287
askfor tx 9258e72d561e900d9f43 0
sending getdata: tx 9258e72d561e900d9f43
AcceptToMemoryPoolUnchecked(): size 25
AcceptToMemoryPool(): accepted 9258e72d56
received getdata for: tx 8d0bb79790fb657abcf5
trying connection 80.109.36.181:8333 lastseen=-2.1hrs
connect() failed after select(): Connection refused
trying connection 75.71.123.118:8333 lastseen=-2.8hrs
askfor tx 92422eb05a06de269337 0
sending getdata: tx 92422eb05a06de269337
askfor tx 92422eb05a06de269337 1333717051000000
askfor tx 92422eb05a06de269337 1333717171000000
askfor tx 92422eb05a06de269337 1333717291000000
askfor tx 92422eb05a06de269337 1333717411000000
askfor tx 92422eb05a06de269337 1333717531000000
askfor tx 92422eb05a06de269337 1333717651000000
askfor tx 92422eb05a06de269337 1333717771000000
askfor tx 92422eb05a06de269337 1333717891000000
askfor tx 92422eb05a06de269337 1333718011000000
askfor tx 92422eb05a06de269337 1333718131000000
askfor tx 92422eb05a06de269337 1333718251000000
askfor tx 92422eb05a06de269337 1333718371000000
AcceptToMemoryPoolUnchecked(): size 26
AcceptToMemoryPool(): accepted 92422eb05a
askfor tx 4119e666abec78284fa8 0
sending getdata: tx 4119e666abec78284fa8
askfor tx 4119e666abec78284fa8 1333717053000000
askfor tx 1c7736165bd767d36f81 0
sending getdata: tx 1c7736165bd767d36f81
AcceptToMemoryPoolUnchecked(): size 27
AcceptToMemoryPool(): accepted 4119e666ab
AcceptToMemoryPoolUnchecked(): size 28
AcceptToMemoryPool(): accepted 1c7736165b
Added 1 addresses from 128.211.220.133: 3415 tried, 8181 new
connection timeout
trying connection 85.134.121.91:8333 lastseen=-2.7hrs
received getdata for: tx bc7df6ba49c3f73c67b5
received getdata for: tx 0824d52714585ec7b346
Added 1 addresses from 174.29.73.225: 3415 tried, 8180 new
askfor tx 4c999081f337a76825bb 0
sending getdata: tx 4c999081f337a76825bb
askfor tx 4c999081f337a76825bb 1333717059000000
askfor tx 4c999081f337a76825bb 1333717179000000
askfor tx 4c999081f337a76825bb 1333717299000000
askfor tx 4c999081f337a76825bb 1333717419000000
askfor tx 4c999081f337a76825bb 1333717539000000
askfor tx 4c999081f337a76825bb 1333717659000000
askfor tx 4c999081f337a76825bb 1333717779000000
askfor tx 4c999081f337a76825bb 1333717899000000
askfor tx 4c999081f337a76825bb 1333718019000000
AcceptToMemoryPoolUnchecked(): size 29
AcceptToMemoryPool(): accepted 4c999081f3
Added 1 addresses from 58.38.114.57: 3415 tried, 8180 new
Added 1 addresses from 86.171.229.45: 3415 tried, 8179 new
connection timeout
trying connection 24.199.159.83:8333 lastseen=-3.6hrs
Added 1 addresses from 71.125.32.246: 3415 tried, 8178 new
Added 1 addresses from 131.93.72.31: 3415 tried, 8179 new
connection timeout
Added 1 addresses from 76.117.220.29: 3415 tried, 8177 new
trying connection 78.159.58.51:8333 lastseen=-3.5hrs
connected 78.159.58.51:8333
Added time data, samples 200, offset -17 (+0 minutes)
Moving 78.159.58.51:8333 to tried
version message: version 32300, blocks=174521
trying connection 95.27.140.103:8333 lastseen=-3.3hrs
Added 8 addresses from 78.159.58.51: 3415 tried, 8170 new
connection timeout
Added 1 addresses from 98.182.22.232: 3415 tried, 8168 new
trying connection 194.226.8.27:8333 lastseen=-2.1hrs
received getdata for: block 0000000000000504f773
ERROR: CBlock::ReadFromDisk() : OpenBlockFile failed
socket closed
disconnecting node 109.75.176.70:8333

...

Code:

Added 1 addresses from 76.117.220.29: 3415 tried, 8128 new
accepted connection 86.47.17.210:10890
Added time data, samples 200, offset -4 (+0 minutes)
Added 86.47.17.210:8333 from 86.47.17.210: 3415 tried, 8127 new
Moving 86.47.17.210:8333 to tried
version message: version 60000, blocks=167425
connection timeout
trying connection 117.22.50.36:8333 lastseen=-2.2hrs
connected 117.22.50.36:8333
trying connection 84.232.230.105:8333 lastseen=-2.0hrs
connect() failed after select(): Connection refused
trying connection 82.170.160.25:8333 lastseen=-2.1hrs
getblocks 166650 to 000000000000027f4096 limit 500
ERROR: CBlock::ReadFromDisk() : OpenBlockFile failed
ERROR: CBlock::ReadFromDisk() : OpenBlockFile failed

.. and now I get literally hundreds of this:

Code:

ERROR: CBlock::ReadFromDisk() : OpenBlockFile failed

.. after which it continues like this:

Code:

ERROR: CBlock::ReadFromDisk() : OpenBlockFile failed
ERROR: CBlock::ReadFromDisk() : OpenBlockFile failed
ERROR: CBlock::ReadFromDisk() : OpenBlockFile failed
getblocks stopping at limit 167149 00000000000003251788 (40500 bytes)
received getdata for: tx 1b356369085657c7ace4
received getdata for: tx 8c50e82b2748528264cf
received getdata for: tx 2198ff40554b152b1a79
received getdata for: tx 828a38228f1721fece77
received getdata for: tx d4a4a2a23faebe9f6e62
received getdata for: tx 227eb66590575d626827
received getdata for: tx f3bc2efde58857ee9e66
received getdata for: tx 905f4ec6f080f66ffd0e
received getdata for: tx cc2fa59268afb6437f27
received getdata for: tx 8204e14270077c5cbf76
received getdata for: tx 5a3e47b43ce378aad707
received getdata for: tx 09a5d50c49714bafeaf8
received getdata for: tx 89c99876c87ccddea130
received getdata for: tx 16b0eedcbe030dc75129
received getdata for: block 0000000000000706fb4a
ERROR: CBlock::ReadFromDisk() : OpenBlockFile failed
received getdata for: tx a26d8f23d50b8ede2221
Added 1 addresses from 213.151.89.23: 3416 tried, 8123 new
connection timeout
socket closed
disconnecting node 68.12.223.12:8333
trying connection 77.93.86.5:8333 lastseen=-9.3hrs
connect() failed after select(): No route to host
trying connection 92.30.18.217:8333 lastseen=-2.7hrs
Added 1 addresses from 24.84.96.223: 3416 tried, 8119 new
Added 1 addresses from 68.53.153.158: 3416 tried, 8120 new
Added 1 addresses from 216.150.78.34: 3416 tried, 8120 new
connection timeout
trying connection 78.42.219.156:8333 lastseen=-2.2hrs
connected 78.42.219.156:8333
trying connection 86.93.165.82:8333 lastseen=-3.2hrs
askfor tx cc18e6dd56b2ac029bb1 0
sending getdata: tx cc18e6dd56b2ac029bb1
askfor tx cc18e6dd56b2ac029bb1 1333717155000000
askfor tx cc18e6dd56b2ac029bb1 1333717275000000
received getdata for: tx 7112939b80c44093c5cc
askfor tx cc18e6dd56b2ac029bb1 1333717395000000
askfor tx cc18e6dd56b2ac029bb1 1333717515000000
ERROR: CTransaction::ReadFromDisk() : OpenBlockFile failed
ERROR: FetchInputs() : cc18e6dd56 ReadFromDisk prev tx 803d7f1f5d failed
ERROR: AcceptToMemoryPool() : FetchInputs failed cc18e6dd56
storing orphan tx cc18e6dd56
Added 1 addresses from 90.214.173.163: 3416 tried, 8115 new
received getdata for: tx a26d8f23d50b8ede2221
received getdata for: tx a26d8f23d50b8ede2221
received getdata for: tx a26d8f23d50b8ede2221
received getdata for: tx a26d8f23d50b8ede2221
received getdata for: tx a26d8f23d50b8ede2221
connection timeout
received getdata for: tx ddb63a21608ada6d5a1d
trying connection 89.223.38.207:8333 lastseen=-4.3hrs
received getdata for: tx a26d8f23d50b8ede2221
received getdata for: tx a26d8f23d50b8ede2221
received getdata for: tx 4febb23f86ab4a0d6cc4
Added 1 addresses from 68.53.153.158: 3416 tried, 8105 new
connection timeout
trying connection 99.244.62.156:8333 lastseen=-4.0hrs
connected 99.244.62.156:8333
Added 1 addresses from 173.238.168.52: 3416 tried, 8106 new
Added 1 addresses from 192.75.95.253: 3416 tried, 8107 new
trying connection 90.129.138.126:8333 lastseen=-2.3hrs
trying connection 216.160.91.91:8333 lastseen=-4.5hrs
trying connection 75.94.171.54:8333 lastseen=-3.4hrs
trying connection 68.207.196.134:8333 lastseen=-2.2hrs
trying connection 80.199.31.74:8333 lastseen=-2.4hrs
trying connection 12.189.154.66:8333 lastseen=-3.1hrs
trying connection 188.47.5.13:8333 lastseen=-2.0hrs
trying connection 46.146.1.200:8333 lastseen=-2.8hrs
trying connection 74.60.78.73:8333 lastseen=-3.2hrs
trying connection 98.222.154.122:8333 lastseen=-2.8hrs
trying connection 77.255.5.179:8333 lastseen=-3.2hrs
socket recv error 110
disconnecting node 109.246.254.188:8333
trying connection 81.151.216.26:8333 lastseen=-4.3hrs
socket closed
disconnecting node 84.73.221.49:8333
terminate called after throwing an instance of 'DbRunRecoveryException'
what(): DbEnv::txn_checkpoint: DB_RUNRECOVERY: Fatal error, run database recovery
Aborted

.. and here bitcoind is dead. No more messages, nor a stackdump either. Resultcode is 134.

I got the same behaviour on two different boxes.

The earliest abnormal behaviour I could find in the scrollback buffer is transaction 7680e65403ab1b258887 (dont know if this identifier is meaningful after bitcoind has shutdown?). Then there is that sole "OpenBlockFile failed", which shortly extends into hundreds of the same error which take down all dependent operations very quickly.

This happened around 7pm CET this afternoon.

If necessary, I think I have full -debug -printtoconsole logs on the other box (not limited by scrollback buffer size). Let me know if you need them and I will recover them for you.

Edit: added code tags