Quote from: jgarzik on April 07, 2012, 08:28:58 AM

run it in gdb, and use the "bt" command when a segfault occurs. Make sure to enable thread tracing, if that is not default / built in.

Ok, so this is a crash with v0.6.0 release at around 6am today. I still dont know how to enable thread tracing, so I hope it was just enable by default (?).

The backbuffer of -debug -printtoconsole looks quite normal:

Code:

Added 1 addresses from 91.154.226.179: 3470 tried, 11409 new
04/08/12 04:06:56 sending: addr (31 bytes)
04/08/12 04:06:57 received: addr (31 bytes)
04/08/12 04:06:57 sending: addr (31 bytes)
04/08/12 04:06:59 sending: addr (31 bytes)
04/08/12 04:07:00 sending: addr (31 bytes)
04/08/12 04:07:00 sending: addr (31 bytes)
04/08/12 04:07:01 sending: addr (31 bytes)
04/08/12 04:07:03 received: inv (37 bytes)
  got inventory: tx 46003e4d1f5d3985d710 new
askfor tx 46003e4d1f5d3985d710 0
sending getdata: tx 46003e4d1f5d3985d710
04/08/12 04:07:03 sending: getdata (37 bytes)
04/08/12 04:07:03 received: inv (37 bytes)
  got inventory: tx 46003e4d1f5d3985d710 new
askfor tx 46003e4d1f5d3985d710 1333858023000000
04/08/12 04:07:03 received: tx (258 bytes)
Rate limit dFreeCount: 13149.2 => 13407.2
AcceptToMemoryPoolUnchecked(): size 132
AcceptToMemoryPool(): accepted 46003e4d1f
04/08/12 04:07:03 received: inv (37 bytes)
  got inventory: tx 46003e4d1f5d3985d710 have
04/08/12 04:07:03 sending: inv (37 bytes)
04/08/12 04:07:03 received: inv (37 bytes)
  got inventory: tx 46003e4d1f5d3985d710 have
04/08/12 04:07:03 sending: inv (37 bytes)
04/08/12 04:07:04 sending: inv (37 bytes)
04/08/12 04:07:04 received: inv (37 bytes)
  got inventory: tx 46003e4d1f5d3985d710 have
04/08/12 04:07:04 received: inv (37 bytes)
  got inventory: tx 46003e4d1f5d3985d710 have
04/08/12 04:07:04 sending: inv (37 bytes)
04/08/12 04:07:04 received: inv (37 bytes)
  got inventory: tx 46003e4d1f5d3985d710 have
04/08/12 04:07:04 sending: inv (37 bytes)
04/08/12 04:07:04 received: inv (37 bytes)
  got inventory: tx 46003e4d1f5d3985d710 have
04/08/12 04:07:04 received: inv (37 bytes)
  got inventory: tx 46003e4d1f5d3985d710 have
04/08/12 04:07:04 sending: inv (37 bytes)
04/08/12 04:07:04 received: inv (37 bytes)
  got inventory: tx 46003e4d1f5d3985d710 have
04/08/12 04:07:04 received: inv (37 bytes)
  got inventory: tx 46003e4d1f5d3985d710 have
04/08/12 04:07:05 sending: inv (37 bytes)
04/08/12 04:07:05 sending: inv (37 bytes)
04/08/12 04:07:05 received: inv (37 bytes)
  got inventory: tx 46003e4d1f5d3985d710 have
04/08/12 04:07:05 received: inv (37 bytes)
  got inventory: tx 46003e4d1f5d3985d710 have
04/08/12 04:07:05 received: inv (37 bytes)
  got inventory: tx 46003e4d1f5d3985d710 have
04/08/12 04:07:05 sending: inv (37 bytes)
04/08/12 04:07:06 received: inv (37 bytes)
  got inventory: tx 46003e4d1f5d3985d710 have
04/08/12 04:07:06 sending: inv (37 bytes)
04/08/12 04:07:07 received: inv (37 bytes)
  got inventory: tx 46003e4d1f5d3985d710 have
04/08/12 04:07:10 received: inv (37 bytes)
  got inventory: tx 46003e4d1f5d3985d710 have
04/08/12 04:07:12 received: addr (31 bytes)
04/08/12 04:07:12 sending: addr (31 bytes)
04/08/12 04:07:16 sending: addr (31 bytes)
04/08/12 04:07:17 received: addr (31 bytes)
04/08/12 04:07:18 sending: addr (31 bytes)
04/08/12 04:07:19 received: addr (31 bytes)
*** glibc detected *** /home/btc/bitcoind: free(): invalid next size (fast): 0x0000000000d9ffa0 ***

Now comes a dump that was made automatically. It looks similar to what I mentioned that had happened already once to me:

Code:

======= Backtrace: =========
[0x7f346b]
[0x7f7516]
[0x40f013]
[0x40f2d8]
[0x40e0ee]
[0x40e3dc]
[0x46d2b4]
[0x470620]
[0x48e435]
[0x48e645]
[0x751edd]
[0x810329]
======= Memory map: ========
00400000-0099b000 r-xp 00000000 fd:02 181991 /home/USERNAME/bitcoind
00b9a000-00bab000 rwxp 0059a000 fd:02 181991 /home/USERNAME/bitcoind
00bab000-038dc000 rwxp 00bab000 00:00 0
038dc000-038dd000 rwxp 038dc000 00:00 0
038dd000-04cb9000 rwxp 038dd000 00:00 0
40000000-40001000 ---p 40000000 00:00 0
40001000-40a01000 rwxp 40001000 00:00 0
40a01000-40a02000 ---p 40a01000 00:00 0
40a02000-41402000 rwxp 40a02000 00:00 0
41402000-41403000 ---p 41402000 00:00 0
41403000-41e03000 rwxp 41403000 00:00 0
41e03000-41e04000 ---p 41e03000 00:00 0
41e04000-42804000 rwxp 41e04000 00:00 0
42804000-42805000 ---p 42804000 00:00 0
42805000-43205000 rwxp 42805000 00:00 0
43205000-43206000 ---p 43205000 00:00 0
43206000-43c06000 rwxp 43206000 00:00 0
43c06000-43c07000 ---p 43c06000 00:00 0
43c07000-44607000 rwxp 43c07000 00:00 0
44607000-44608000 ---p 44607000 00:00 0
44608000-45008000 rwxp 44608000 00:00 0
45008000-45009000 ---p 45008000 00:00 0
45009000-45a09000 rwxp 45009000 00:00 0
2aaaaaaab000-2aaaaaaae000 r-xp 2aaaaaaab000 00:00 0 [vdso]
2aaaaaaae000-2aaaae07c000 r-xp 00000000 fd:00 4475090 /usr/lib/locale/locale-archive
2aaaae07c000-2aaaae083000 r-xs 00000000 fd:00 8786 /usr/lib64/gconv/gconv-modules.cache
2aaaae083000-2aaaae085000 rwxp 2aaaae083000 00:00 0
2aaaae085000-2aaaae08b000 rwxs 00000000 fd:02 37965938 /home/USERNAME/.bitcoin/__db.001
2aaaae08b000-2aaaae281000 rwxs 00000000 fd:02 37965939 /home/USERNAME/.bitcoin/__db.002
2aaaae281000-2aaab01c3000 rwxs 00000000 fd:02 37965940 /home/USERNAME/.bitcoin/__db.003
2aaab01c3000-2aaab02e3000 rwxs 00000000 fd:02 37965941 /home/USERNAME/.bitcoin/__db.004
2aaab02e3000-2aaab08e9000 rwxs 00000000 fd:02 37965942 /home/USERNAME/.bitcoin/__db.005
2aaab08e9000-2aaab08f5000 rwxs 00000000 fd:02 37965943 /home/USERNAME/.bitcoin/__db.006
2aaab08fb000-2aaab0905000 r-xp 00000000 09:01 130597 /lib64/libnss_files-2.5.so
2aaab0905000-2aaab0b04000 ---p 0000a000 09:01 130597 /lib64/libnss_files-2.5.so
2aaab0b04000-2aaab0b05000 r-xp 00009000 09:01 130597 /lib64/libnss_files-2.5.so
2aaab0b05000-2aaab0b06000 rwxp 0000a000 09:01 130597 /lib64/libnss_files-2.5.so
2aaab0b06000-2aaab0c53000 r-xp 00000000 09:01 130565 /lib64/libc-2.5.so
2aaab0c53000-2aaab0e53000 ---p 0014d000 09:01 130565 /lib64/libc-2.5.so
2aaab0e53000-2aaab0e57000 r-xp 0014d000 09:01 130565 /lib64/libc-2.5.so
2aaab0e57000-2aaab0e58000 rwxp 00151000 09:01 130565 /lib64/libc-2.5.so
2aaab0e58000-2aaab0e5d000 rwxp 2aaab0e58000 00:00 0
2aaab0e5d000-2aaab0e79000 r-xp 00000000 09:01 130587 /lib64/ld-2.5.so
2aaab0e79000-2aaab1079000 ---p 0001c000 09:01 130587 /lib64/ld-2.5.so
2aaab1079000-2aaab107a000 r-xp 0001c000 09:01 130587 /lib64/ld-2.5.so
2aaab107a000-2aaab107b000 rwxp 0001d000 09:01 130587 /lib64/ld-2.5.so
2aaab107b000-2aaab117b000 rwxp 2aaab107b000 00:00 0
2aaab117b000-2aaab117f000 r-xp 00000000 09:01 130596 /lib64/libnss_dns-2.5.so
2aaab117f000-2aaab137e000 ---p 00004000 09:01 130596 /lib64/libnss_dns-2.5.so
2aaab137e000-2aaab137f000 r-xp 00003000 09:01 130596 /lib64/libnss_dns-2.5.so
2aaab137f000-2aaab1380000 rwxp 00004000 09:01 130596 /lib64/libnss_dns-2.5.so
2aaab1380000-2aaab1391000 r-xp 00000000 09:01 130606 /lib64/libresolv-2.5.so
2aaab1391000-2aaab1591000 ---p 00011000 09:01 130606 /lib64/libresolv-2.5.so
2aaab1591000-2aaab1592000 r-xp 00011000 09:01 130606 /lib64/libresolv-2.5.so
2aaab1592000-2aaab1593000 rwxp 00012000 09:01 130606 /lib64/libresolv-2.5.so
2aaab1593000-2aaab1595000 rwxp 2aaab1593000 00:00 0
2aaab2117000-2aaab272f000 rwxp 2aaab2117000 00:00 0
2aaab4000000-2aaab5e3d000 rwxp 2aaab4000000 00:00 0
2aaab5e3d000-2aaab8000000 ---p 2aaab5e3d000 00:00 0
7ffffffea000-7ffffffff000 rwxp 7ffffffe9000 00:00 0 [stack]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vsyscall]

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x43c05940 (LWP 1393)]
0x000000000082ce95 in raise ()

When I use the bt command, I get this:

Code:

#0 0x000000000082ce95 in raise ()
#1 0x00000000007cfb10 in abort ()
#2 0x00000000007eaaeb in __libc_message ()
#3 0x00000000007f346b in _int_free ()
#4 0x00000000007f7516 in free ()
#5 0x000000000040f013 in erase (this=0xbab918, __first=, __last=...)
   at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/ext/new_allocator.h:94
#6 std::_Rb_tree, std::_Select1st >, std::less, std::allocator > >::erase (this=0xbab918, __first=, __last=...)
   at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_tree.h:1281
#7 0x000000000040f2d8 in std::_Rb_tree, std::_Select1st >, std::less, std::allocator > >::erase (this=0xbab918, __x=)
   at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_tree.h:1215
#8 0x000000000040e0ee in erase (this=0xbab8a0, nUBucket=)
   at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_map.h:461
#9 CAddrMan::ShrinkNew (this=0xbab8a0, nUBucket=) at addrman.cpp:181
#10 0x000000000040e3dc in CAddrMan::Add_ (this=0xbab8a0, addr=, source=, nTimePenalty=)
   at addrman.cpp:346
#11 0x000000000046d2b4 in ProcessMessage (pfrom=0x2aaab4bb0450, strCommand=..., vRecv=...) at addrman.h:433
#12 0x0000000000470620 in ProcessMessages (pfrom=0x2aaab4bb0450) at main.cpp:2767
#13 0x000000000048e435 in ThreadMessageHandler2 (parg=) at net.cpp:1516
#14 0x000000000048e645 in ThreadMessageHandler (parg=0x0) at net.cpp:1481
#15 0x0000000000751edd in start_thread ()
#16 0x0000000000810329 in clone ()

I have to leave for a few hours, but will keep the debugger open. If there are any commands that can extract more useful information from it, please post here and I will run them on it tonight.

Edit: Added code tags (good idea). BTW, the box is still up, for if anyone wants to see the output of another gdb command, memory dump, etc.