與符號表分離程序或動態庫, 如何用GDB調試
五竹
1 ? ? ? Debugging Information in Separate Files
GDB支持用戶將程序調試信息放在一個獨立的文件里，而不是和可執行程序在一個文件中，GDB可以某種方式來查找和自動加載調試信息。由于調試信息可能非常大,有時可能比可執行代碼自身還要大,某些系統將其可執行程序的調試信息以單獨的文件發布，在需要調試問題的時候，用戶可以再安裝這些文件。
GDB支持兩種設置單獨調試信息文件的方式：
可執行程序里包含了一個調試鏈接，此鏈接指定了單獨的調試信息文件名。單獨調試文件名通常是’executable.debug’， executable是相應的可執行程序名，不帶路徑（例如，’ls.debug’是’/usr/bin/ls’的調試信息文件）。此外，調試鏈接為調試 文件設置了CRC32的校驗和，GDB用此校驗和來確?？蓤绦形募驼{試文件是同一個版本的。
可執行文件包含一個版本ID號和一個唯一的bit串，而相應的調試信息文件也包含此bit串。（此方式只在某些系統上支持，特別是那些在二進制文件里使用ELF格式和GNU Binutils的系統。）更多關于此功能的細節，參見’–build-id’命令行選項的介紹 ，在GNU連接器的“命令行選項”節中。雖然版本ID號沒有執行指出調試信息文件名，但是可以從版本ID號里計算出來，參見下 面。
由于有兩種方法可以設置調試信息文件，GDB也用兩種不同的方式查找調試信息文件：
對于“調試鏈接”方式，GDB在可執行文件的目錄里查找對應名字的文件，接著在此目錄下的子目錄’.debug’下查找，最后在全局調試目錄下的一個子目錄里查找，此子目錄的名字和可執行文件的絕對文件名的先導目錄名相同。
對于“版本ID”方式，GDB在全局調試目錄下的’.build-id’子目錄下查找名為’nn/nnnnnnnn.debug’的文件，這里nn是版本ID字符串的頭兩個16進制字符，nnnnnnnn是余下的字符。（真正的版本ID字符串是32個或更多的16進制字符，不是10個。）
舉個例子，假設你要調試’/usr/bin/ls’,此程序有個調試鏈接指定了調試文件’ls.debug’，且其版本ID是是16進制的abcdef1234。如果全局調試目錄是’/usr/lib/debug’，那么GDB按順序查找下列調試信息文件：
–‘/usr/lib/debug/.build-id/ab/cdef1234.debug’
–‘/usr/bin/ls.debug’
–‘/usr/bin/.debug/ls.debug’
–‘/usr/lib/debug/usr/bin/ls.debug’.
你可以設置全局調試信息目錄的名稱，并查看當前GDB所使用的名稱。
set debug-file-directory directory
將directory設置為GDB搜索單獨調試信息文件的目錄。
show debug-file-directory
顯示搜索單獨調試信息文件的目錄。
2 ? ? ? gdb 調試與符號表分離的二進制程序
一般情況下,如果沒有在程序的 spec 中明確指定不進行 strip,則缺省打rpm包時,都會把二進制程序或動態庫的符號表等 debuginfo 信息與執行程序分離,生成一個debuginfo 的 rpm 包. 如 localagent 打 rpm 包時,會生成如下 3 個rpm 包:
local_agent-0.7.1-rc_1.x86_64.rpm
local_agent-debuginfo-0.7.1-rc_1.x86_64.rpm
local_agent-devel-0.7.1-rc_1.x86_64.rpm
將 local_agent-debuginfo-0.7.1-rc_1.x86_64.rpm 解壓,可以看到相應的 debug info 文件.
[tmp]$ rpm2cpio local_agent-debuginfo-0.7.1-rc_1.x86_64.rpm |cpio –idv
./usr/local/bin/.debug/local_agent_client.debug
./usr/local/bin/.debug/local_agent_server.debug
./usr/local/lib64/.debug/liblocal_agent.so.debug
./usr/src/debug/local_agent-0.7.1
./usr/src/debug/local_agent-0.7.1/build
./usr/src/debug/local_agent-0.7.1/build/release64
./usr/src/debug/local_agent-0.7.1/build/release64/local_agent
./usr/src/debug/local_agent-0.7.1/build/release64/local_agent/ClientParamParser.cpp
./usr/src/debug/local_agent-0.7.1/build/release64/local_agent/FileUtil.cpp
./usr/src/debug/local_agent-0.7.1/build/release64/local_agent/LocalAgent.cpp
./usr/src/debug/local_agent-0.7.1/build/release64/local_agent/LocalAgentClient.cpp
./usr/src/debug/local_agent-0.7.1/build/release64/local_agent/LocalAgentClientMain.cpp
./usr/src/debug/local_agent-0.7.1/build/release64/local_agent/LocalAgentMain.cpp
./usr/src/debug/local_agent-0.7.1/build/release64/local_agent/LocalAgentServerAdapter.cpp
./usr/src/debug/local_agent-0.7.1/build/release64/local_agent/ServerParamParser.cpp
./usr/src/debug/local_agent-0.7.1/build/release64/local_agent/SystemWrapper.cpp
./usr/src/debug/local_agent-0.7.1/build/release64/local_agent/local_agent.pb.cc
./usr/src/debug/local_agent-0.7.1/build/release64/local_agent/local_agent.pb.h
./usr/src/debug/local_agent-0.7.1/local_agent
./usr/src/debug/local_agent-0.7.1/local_agent/AgentErrorDefine.h
./usr/src/debug/local_agent-0.7.1/local_agent/ClientParamParser.h
./usr/src/debug/local_agent-0.7.1/local_agent/FileUtil.h
./usr/src/debug/local_agent-0.7.1/local_agent/LocalAgent.h
./usr/src/debug/local_agent-0.7.1/local_agent/LocalAgentClient.h
./usr/src/debug/local_agent-0.7.1/local_agent/LocalAgentServerAdapter.h
./usr/src/debug/local_agent-0.7.1/local_agent/ServerParamParser.h
./usr/src/debug/local_agent-0.7.1/local_agent/SystemWrapper.h
對于這種符號表與二進程序分離的程序,該如何調試呢? 先看下例: ? [local]$ gdb bin/local_agent_server
GNU gdb Fedora (6.8-37.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. ?Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
(no debugging symbols found)
(gdb) b main
Breakpoint 1 at 0x401a80
(gdb) r
Starting program: /home/admin/tmp/usr/local/bin/local_agent_server
[Thread debugging using libthread_db enabled]
[New Thread 0x2ad3e6935860 (LWP 27869)]
Breakpoint 1, 0x0000000000401a80 in main ()
(gdb) bt
#0 ?0x0000000000401a80 in main ()
(gdb)
上述例子可以看出,因為gdb 沒有符號表,所以顯示的都是二進制地址.
2.1 ?方法一: gdb 啟動時通過 –s 指定
對于這種符號表與二進程序,可以在gdb 啟動時,通過 –s 指定符號表文件來解決. 如下例,通過 –s 指定debug/local_agent_server.debug 后, gdb 調試時可以看到符號表了.
[ local]$ gdb -e bin/local_agent_server -s debug/local_agent_server.debug
GNU gdb Fedora (6.8-37.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. ?Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
(gdb) b main
Breakpoint 1 at 0x401a80: file build/release64/local_agent/LocalAgentMain.cpp, line 34.
(gdb) bt
No stack.
(gdb) r
Starting program: /home/admin/tmp/usr/local/bin/local_agent_server
[Thread debugging using libthread_db enabled]
[New Thread 0x2af3399a7860 (LWP 27862)]
Breakpoint 1, main (argc=1, argv=0x7fff5728f0b8) at build/release64/local_agent/LocalAgentMain.cpp:34
34 ? ? ?build/release64/local_agent/LocalAgentMain.cpp: No such file or directory.
in build/release64/local_agent/LocalAgentMain.cpp
(gdb) quit
如果是這種符號表與二進程序分離的程序進行所產生的 core ,可以通過如下方式調試:
gdb -c core.1234 -e bin/local_agent_server -s debug/local_agent_server.debug
其中：
-c 指定core文件
-e 指定binary，用線上的binary即可
-s 指定符號表，也就是我們新生成的符號表
2.2 ?方法二: 將 .debug 目錄放到可執行程序所在的目錄
除了通過 –s 指定 debuginfo 文件外,還可以將 .debug 目錄復制到可執行程序所在的目錄或者將 local_agent_server.debug 復制到可執行程序所在的目錄.
[admin@s002182.cm6 local]$ pwd ?/home/admin/tmp/usr/local
[admin@s002182.cm6 local]$ cp -r debug/ bin/.debug
[admin@s002182.cm6 local]$ ls -lha bin/
total 44K
drwxr-xr-x 3 admin admin 4.0K Mar 21 18:41 .
drwxr-xr-x 6 admin admin 4.0K Mar 21 17:47 ..
drwxr-xr-x 3 admin admin 4.0K Mar 21 18:49 .debug
-rwxr-xr-x 1 admin admin ?14K Mar 21 17:44 local_agent_client
-rwxr-xr-x 1 admin admin ?14K Mar 21 17:44 local_agent_server
此時,也可以看到符號表了.
[local]$ gdb bin/local_agent_server
GNU gdb Fedora (6.8-37.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. ?Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
(gdb) b main
Breakpoint 1 at 0x401a80: file build/release64/local_agent/LocalAgentMain.cpp, line 34.
(gdb) q
2.3 ?Set debug-file-directory 方式不生效
嘗試用 set debug-file-directory 方式來設置 debug 路徑,好像不生效.
[admin@s002182.cm6 local]$ gdb bin/local_agent_server
GNU gdb Fedora (6.8-37.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. ?Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
(no debugging symbols found)
(gdb) b main
Breakpoint 1 at 0x401a80
(gdb) show debug-file-directory
The directory where separate debug symbols are searched for is "/usr/lib/debug".
(gdb) set debug-file-directory /usr/lib/debug:/home/admin/tmp/usr/local/debug
(gdb) show debug-file-directory
The directory where separate debug symbols are searched for is "/usr/lib/debug:/home/admin/tmp/usr/local/debug".
(gdb) shell ls -lh /home/admin/tmp/usr/local/debug
total 416K
-rwxr-xr-x 1 admin admin 203K Mar 21 17:44 local_agent_client.debug
-rwxr-xr-x 1 admin admin 201K Mar 21 17:44 local_agent_server.debug
(gdb) r
Starting program: /home/admin/tmp/usr/local/bin/local_agent_server
(no debugging symbols found)
[Thread debugging using libthread_db enabled]
[New Thread 0x2b5703177860 (LWP 29389)]
Breakpoint 1, 0x0000000000401a80 in main ()
(gdb)
3 ? ? ? gdb 調試與符號表分離的動態庫
如果二進制程序所依賴的動態庫是符號表等調試信息與動態庫二進制分離的,那又該如何讓 gdb 加載這些動態庫的符號表呢?
3.1.1 ? 可執行程序運行時,依賴的動態庫是通過 rpm 包安裝的情況
這種比較簡單,只要將 debuginfo 的 rpm 包將通過 rpm 安裝即可,因為.debug 目錄缺省會安裝在 動態庫所在的目錄下. 下面的 anet 庫的rpm 包解壓后,可以清楚的看到這一點.
[admin@s002182.cm6 tmp]$ rpm2cpio anet-1.3.2-rc_2.x86_64.rpm |cpio -idv
./usr/local/lib64/libanet.so
./usr/local/lib64/libanet.so.3
./usr/local/lib64/libanet.so.3.0.0
563 blocks
[admin@s002182.cm6 tmp]$ rpm2cpio anet-debuginfo-1.3.2-rc_2.x86_64.rpm |cpio -idv
./usr/local/lib64/.debug/libanet.so.3.0.0.debug
./usr/src/debug/anet-1.3.2
./usr/src/debug/anet-1.3.2/anet
./usr/src/debug/anet-1.3.2/anet/addrspec.h
./usr/src/debug/anet-1.3.2/anet/adminclient.h
./usr/src/debug/anet-1.3.2/anet/admincmds.h
./usr/src/debug/anet-1.3.2/anet/adminserver.h
./usr/src/debug/anet-1.3.2/anet/advancepacket.h
./usr/src/debug/anet-1.3.2/anet/advancepacketfactory.h
./usr/src/debug/anet-1.3.2/anet/anet.h
./usr/src/debug/anet-1.3.2/anet/aneterror.h
./usr/src/debug/anet-1.3.2/anet/appadapter.h
3.1.2 ? 可執行程序運行時,依賴的動態庫是通過 rpm2cipo 解壓到某個目錄的情況
如果可執行程序運行時,依賴的動態庫是用戶自已通過 rpm2cpio 解壓后復制到某個用戶自定義的目錄的,那么,拿到該動態庫的 debuginfo rpm 后,也相應的用 rpm2cpio 解壓,并把 .debug 目錄復制到相應的動態庫所在的目錄.
如下面sap_server的示例, sap_server 依賴 libarpc.so.1,但 libarpc.so.1 在打是將 rpm 包時, 其debuginfo 信息分離在單獨的 debuginfo rpm 包中的. 所以導致 gdb core時,看不到 libarpc.so.1 的符號表.
gdb bin/sap_server_d core.17131
(gdb) bt
#0 ?0x0000003959230265 in raise (sig=) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 ?0x0000003959231d10 in abort () at abort.c:88
#2 ?0x000000395d6bec44 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:97
#3 ?0x000000395d6bcdb6 in __cxxabiv1::__terminate (handler=)
at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:43
#4 ?0x000000395d6bcde3 in std::terminate () at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:53
#5 ?0x000000395d6bd2ef in __cxa_pure_virtual () at ../../../../libstdc++-v3/libsupc++/pure.cc:55
#6 ?0x00002b8729bfc933 in arpc::ClientPacketHandler::handlePacket () from /home/admin/sap/lib/libarpc.so.1
#7 ?0x00002b872a0ddcd8 in anet::Connection::handlePacket () from /home/admin/sap/lib/libanet.so.3
#8 ?0x00002b872a0e75fb in anet::TCPConnection::readData () from /home/admin/sap/lib/libanet.so.3
#9 ?0x00002b872a0e624b in anet::TCPComponent::handleReadEvent () from /home/admin/sap/lib/libanet.so.3
#10 0x00002b872a0e8c0b in anet::Transport::eventIteration () from /home/admin/sap/lib/libanet.so.3
#11 0x00002b872a0e8cf2 in anet::Transport::eventLoop () from /home/admin/sap/lib/libanet.so.3
#12 0x00002b872a0e8d47 in anet::Transport::run () from /home/admin/sap/lib/libanet.so.3
#13 0x00002b872a0ea02d in anet::Thread::hook () from /home/admin/sap/lib/libanet.so.3
#14 0x0000003959e064a7 in start_thread (arg=) at pthread_create.c:297
#15 0x00000039592d3c2d in clone () from /lib64/libc.so.6
(gdb) thread 2
要讓 gdb 調試 core 時,能顯示 動態庫的符號表,可以用如下方法:
a). 先解壓 debuginfo ?rpm 包
1
rpm2cpio arpc-debuginfo-0.14.1-rc_1.x86_64.rpm |cipo -idv
b). ?將 ./usr/local/lib64/.debug 復制到 ?libarpc.so.1 所在目錄:
1
/home/admin/sap/lib/ ?cp -r ?./usr/local/lib64/.debug ?/home/admin/sap/lib/
[admin@s007238.cm6 sap]$ gdb bin/sap_server_d core.17131
GNU gdb Fedora (6.8-37.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. ?Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
Reading symbols from /home/admin/sap/lib/libhb_node.so...done.
Loaded symbols for /home/admin/sap/lib/libhb_node.so
Reading symbols from /home/admin/sap/lib/libcm_basic.so...done.
Loaded symbols for /home/admin/sap/lib/libcm_basic.so
Reading symbols from /home/admin/sap/lib/libarpc.so.1...Reading symbols from /home/admin/sap/lib/.debug/libarpc.so.1.0.0.debug...done.
... ...
#0 ?0x0000003959230265 in raise (sig=) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64 ? ? ? ?return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt
#0 ?0x0000003959230265 in raise (sig=) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 ?0x0000003959231d10 in abort () at abort.c:88
#2 ?0x000000395d6bec44 in __gnu_cxx::__verbose_terminate_handler () at ../../../../libstdc++-v3/libsupc++/vterminate.cc:97
#3 ?0x000000395d6bcdb6 in __cxxabiv1::__terminate (handler=)
at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:43
#4 ?0x000000395d6bcde3 in std::terminate () at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:53
#5 ?0x000000395d6bd2ef in __cxa_pure_virtual () at ../../../../libstdc++-v3/libsupc++/pure.cc:55
#6 ?0x00002b8729bfc933 in arpc::ClientPacketHandler::handlePacket (this=0x2aac6df6ee38, packet=0x2aac6df6fc30,
args=) at build/release64/arpc/ClientPacketHandler.cpp:35
#7 ?0x00002b872a0ddcd8 in anet::Connection::handlePacket () from /home/admin/sap/lib/libanet.so.3
#8 ?0x00002b872a0e75fb in anet::TCPConnection::readData () from /home/admin/sap/lib/libanet.so.3
#9 ?0x00002b872a0e624b in anet::TCPComponent::handleReadEvent () from /home/admin/sap/lib/libanet.so.3
#10 0x00002b872a0e8c0b in anet::Transport::eventIteration () from /home/admin/sap/lib/libanet.so.3
#11 0x00002b872a0e8cf2 in anet::Transport::eventLoop () from /home/admin/sap/lib/libanet.so.3
#12 0x00002b872a0e8d47 in anet::Transport::run () from /home/admin/sap/lib/libanet.so.3
#13 0x00002b872a0ea02d in anet::Thread::hook () from /home/admin/sap/lib/libanet.so.3
#14 0x0000003959e064a7 in start_thread (arg=) at pthread_create.c:297
#15 0x00000039592d3c2d in clone () from /lib64/libc.so.6
Current language: ?auto; currently c
