一、breakpad簡介
breakpad 而是一個全平臺的C/C++程序的崩潰日志收集工具，適配了Windows/MacOX/Linux，當然也支持了Android。
breadpad的工作原理如下圖所示:
image.png
breadpad 主要包含三個模塊：
client
編譯進入項目中，隨項目一起編譯發布，發布出去的so是strip掉debug信息的。當在用戶手機上崩潰的時候，client就收集信息，寫入特定格式的崩潰文件。文件最后被收集到服務端。Breakpad symbol dumper （dump_syms工具）
當你在編譯so的時候，除了編譯strip后的so，還得保留strip前的so。dump_syms 就是用來從strip前的so 提取符號信息.sym文件
如何獲得保留strip信息的so?
假如你是用gradle編譯的話，strip前的so和strip后的so都是存在的，可以在特定目錄查找,如下：
└── transforms
├── mergeJniLibs
│ └── debug
│ ├── 0
│ │ └── lib
│ │ └── arm64-v8a
│ │ └── libbreakpad-core.so
│ └── __content__.json
└── stripDebugSymbol
└── debug
├── 0
│ └── lib
│ └── arm64-v8a
│ └── libbreakpad-core.so
└── __content__.json
app/build/intermediates/transforms/ 目錄下，mergeJniLibs中的so是strip debug信息之前的so;stripDebugSymbol中的so是strip掉debug信息之后的so
找到未strip的so之后，可以用以下命令提取出符號信息
./dump_syms libbreakpad-core.so > libbreakpad-core.so.sy
- minidump processer（minidump_stackwalk工具）
sym符號文件和.dmp minidump文件發送給Server端后，通過minidump_stackwalk指令，從.sym符號文件和包含崩潰信息的.dmp文件中提取出完整的奔潰時的堆棧信息。
二、下載breakpad源碼
可以從breadpad-github，然后編譯。
很人反映從官網下載的breadpad源碼編譯會報錯，需要補充缺失的文件。
所以也可以直接
feifei-123的github直接下載可以編譯通過的版本。
執行命令:
git clone https://github.com/feifei-123/breakpad
三、編譯breadpad
breakpad 是跨平臺的，支持linux、window和Mac os系統，不同平臺上的編譯配置也是不同的。
- linux 平臺編譯出來的dump_syms 僅能再linux上運行，來解析linux上運行的so的符號信息
- macOS 平臺編譯出來的dump_syms 僅能再mac OS 上運行，來解析mac 上運行的so的符號信息。
- window 平臺編譯出來的dump_syms，僅能在Window上運行，并解析window上運行的dll的符號信息。
因為我最終的應用場景時 捕捉Android上的natvie crash，然后解析奔潰時的堆棧，而Android 是屬于Linux系統的。
所以編譯breadpad 也必須在Linux系統上進行。否則編譯出來的dump_syms 無法解析android的minidump信息的。
編譯過程:
清空上次build的臨時文件
make clean
運行配置信息
./configure
parallels@parallels-vm:~/Desktop/Parallels Shared Folders/Home/Desktop/TM/Github/orginal_breakpad/breakpad$ ./configure
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether UID '1000' is supported by ustar format... yes
checking whether GID '1000' is supported by ustar format... yes
checking how to create a ustar tar archive... gnutar
checking whether to enable maintainer-specific portions of Makefiles... no
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... gcc3
checking for ar... ar
checking the archiver (ar) interface... ar
checking dependency style of gcc... gcc3
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking whether gcc understands -c and -o together... (cached) yes
checking dependency style of gcc... (cached) gcc3
checking how to run the C preprocessor... gcc -E
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking dependency style of g++... gcc3
checking for ranlib... ranlib
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for the pthreads library -lpthreads... no
checking whether pthreads work without any flags... no
checking whether pthreads work with -Kthread... no
checking whether pthreads work with -kthread... no
checking for the pthreads library -llthread... no
checking whether pthreads work with -pthread... yes
checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE
checking if more special flags are required for pthreads... no
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking a.out.h usability... yes
checking a.out.h presence... yes
checking for a.out.h... yes
checking sys/random.h usability... no
checking sys/random.h presence... no
checking for sys/random.h... no
checking for arc4random... no
checking for getrandom... no
checking whether g++ supports C++11 features by default... no
checking whether g++ supports C++11 features with -std=c++11... yes
checking whether C++ compiler accepts -Werror=unknown-warning-option... no
checking whether C++ compiler accepts -Wmissing-braces... yes
checking whether C++ compiler accepts -Wnon-virtual-dtor... yes
checking whether C++ compiler accepts -Woverloaded-virtual... yes
checking whether C++ compiler accepts -Wreorder... yes
checking whether C++ compiler accepts -Wsign-compare... yes
checking whether C++ compiler accepts -Wunused-local-typedefs... yes
checking whether C++ compiler accepts -Wunused-variable... yes
checking whether C++ compiler accepts -Wvla... yes
checking for O_CLOEXEC defined in fcntl.h... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating breakpad.pc
config.status: creating breakpad-client.pc
config.status: creating Makefile
config.status: creating src/config.h
config.status: src/config.h is unchanged
config.status: executing depfiles commands
注意 checking build system type... x86_64-pc-linux-gnu 表明我當前的編譯環境是linux
make 執行編譯
make
編譯完成后
libbreakpad_client.a 位于/src/client/linux 目錄下，此文件可以編譯進android 的app,完成native crash的捕捉和生成minidump文件
dump_syms 文件位于 /src/tools/linux/dump_syms 目錄，用于提取so庫的sym符號文件
parallels@parallels-vm:~/Desktop/Parallels Shared Folders/Home/Desktop/TM/Github/orginal_breakpad/breakpad/src/tools/linux/dump_syms$ ls
dump_syms dump_syms.cc src_tools_linux_dump_syms_dump_syms-dump_syms.o
- minidump_stackwalk 位于/src/processor目錄下，用于將.dmp minudump文件和.sym文件合成可讀的堆棧信息
parallels@parallels-vm:~/Desktop/Parallels Shared Folders/Home/Desktop/TM/Github/orginal_breakpad/breakpad/src/processor$ ls | grep minidump_stackwalk
minidump_stackwalk
minidump_stackwalk.cc
minidump_stackwalk_machine_readable_test
minidump_stackwalk.o
minidump_stackwalk_test
三、解析minudump文件
1、 dump_syms 提取特定so庫的符號信息
以libbreakpad-core.so為例：
./dump_syms libbreakpad-core.so > libbreakpad-core.so.sym
2、根據1中生成的libbreakpad-core.so.sym生成特定的目錄結構：
├── symbol
│ └── libbreakpad-core.so
│ └── 57399AA1EE2607A34686D5DED7D43C310
│ └── libbreakpad-core.so.sym
命令如下:
head -n1 libbreakpad-core.so.sym
MODULE Linux arm64 57399AA1EE2607A34686D5DED7D43C310 libbreakpad-core.so
mkdir -p ./symbol/libbreakpad-core.so/57399AA1EE2607A34686D5DED7D43C310
mv libbreakpad-core.so.sym ./symbol/libbreakpad-core.so/57399AA1EE2607A34686D5DED7D43C310/
3、調用minidump_stackwalk命令，將dmp文件和sym文件合成可讀的crashinfo.txt
./minidump_stackwalk 8439c979-cecf-41a6-25eaf89c-dbf9c03b.dmp ./symbol > crashinfo.txt
crashinfo.txt 部分內容如下:
Operating system: Android
0.0.0 Linux 4.4.83 #2 SMP PREEMPT Sun Jan 12 10:48:20 CST 2020 aarch64
CPU: arm64
4 CPUs
GPU: UNKNOWN
Crash reason: SIGSEGV /SEGV_MAPERR
Crash address: 0x0
Process uptime: not available
Thread 32 (crashed)
0 libbreakpad-core.so!testCrash1() + 0x14
x0 = 0x0000000000000027 x1 = 0x00000078e60f5f30
x2 = 0x0000000000000005 x3 = 0x0000000000000003
x4 = 0x0000000040100401 x5 = 0xa800000040404000
x6 = 0x0000000000000000 x7 = 0x7f7f7f7f7f7f7f7f
x8 = 0x0000000000000000 x9 = 0x0000000000000001
x10 = 0x00000078e60f60c0 x11 = 0x0000000000000018
x12 = 0x000000000000000b x13 = 0xffffffffffffffff
x14 = 0xff00000000000000 x15 = 0xffffffffffffffff
x16 = 0x00000078e82b7608 x17 = 0x00000078e823b454
x18 = 0x0000000000000008 x19 = 0x00000078e7e2fe00
x20 = 0x0000007902ae7f20 x21 = 0x00000078e7e2fe00
x22 = 0x00000078e60f693c x23 = 0x00000078e8f1bc31
x24 = 0x0000000000000000 x25 = 0x00000078e60f7588
x26 = 0x00000078e7e2fea0 x27 = 0x0000000000000000
x28 = 0x0000000000000000 fp = 0x00000078e60f6650
lr = 0x00000078e823b4c8 sp = 0x00000078e60f6620
pc = 0x00000078e823b468
Found by: given as instruction pointer in context
1 libbreakpad-core.so!testCoffeCacher() + 0x4c
x19 = 0x00000078e7e2fe00 x20 = 0x0000007902ae7f20
x21 = 0x00000078e7e2fe00 x22 = 0x00000078e60f693c
x23 = 0x00000078e8f1bc31 x24 = 0x0000000000000000
x25 = 0x00000078e60f7588 x26 = 0x00000078e7e2fea0
x27 = 0x0000000000000000 x28 = 0x0000000000000000
fp = 0x00000078e60f6650 sp = 0x00000078e60f6630
pc = 0x00000078e823b4c8
Found by: call frame info
2 libbreakpad-core.so!call_dangerous_function() + 0x14
x19 = 0x00000078e7e2fe00 x20 = 0x0000007902ae7f20
x21 = 0x00000078e7e2fe00 x22 = 0x00000078e60f693c
x23 = 0x00000078e8f1bc31 x24 = 0x0000000000000000
x25 = 0x00000078e60f7588 x26 = 0x00000078e7e2fea0
x27 = 0x0000000000000000 x28 = 0x0000000000000000
fp = 0x00000078e60f6670 sp = 0x00000078e60f6660
pc = 0x00000078e823b508
Found by: call frame info
3 libbreakpad-core.so!Java_com_sogou_translate_jni_BreakpadInit_go2crash + 0x14
x19 = 0x00000078e7e2fe00 x20 = 0x0000007902ae7f20
x21 = 0x00000078e7e2fe00 x22 = 0x00000078e60f693c
x23 = 0x00000078e8f1bc31 x24 = 0x0000000000000000
x25 = 0x00000078e60f7588 x26 = 0x00000078e7e2fea0
x27 = 0x0000000000000000 x28 = 0x0000000000000000
fp = 0x00000078e60f6690 sp = 0x00000078e60f6680
pc = 0x00000078e823b534
Found by: call frame info
4 base.odex + 0x111c0
x19 = 0x00000078e7e2fe00 x20 = 0x0000007902ae7f20
x21 = 0x00000078e7e2fe00 x22 = 0x00000078e60f693c
x23 = 0x00000078e8f1bc31 x24 = 0x0000000000000000
x25 = 0x00000078e60f7588 x26 = 0x00000078e7e2fea0
x27 = 0x0000000000000000 x28 = 0x0000000000000000
fp = 0x00000078e60f6768 sp = 0x00000078e60f66a0
pc = 0x00000078e8e261c4
Found by: call frame info
四、一路踩的坑
因為我自己的筆記本是mac pro,最初編譯breakpad是在mac上編譯的。
執行./configure & make 成功編譯之后
./configure & make
feifeideMacBook-Pro:src feifei$ ./configure & make
[1] 45277
make: *** No targets specified and no makefile found. Stop.
feifeideMacBook-Pro:src feifei$ checking build system type... x86_64-apple-darwin18.5.0
checking host system type... x86_64-apple-darwin18.5.0
在/src/client/linux中未生成 dump_syms文件,
feifeideMacBook-Pro:dump_syms feifei$ pwd
/Users/feifei/Desktop/tmp/breakpad/src/src/tools/linux/dump_syms
feifeideMacBook-Pro:dump_syms feifei$ ls
dump_syms.cc
進而去/src/client/mac中 存在一個dump_syms.xcodeproj文件。
/Users/feifei/Desktop/tmp/breakpad/src/src/tools/mac/dump_syms
feifeideMacBook-Pro:dump_syms feifei$ ls
dump_syms.xcodeproj dump_syms_tool.cc macho_dump.cc
利用xcode 打開dump_syms.xcodeproj編譯通過
image.png
得到dump_syms,利用該dump_syms 解析Android 工程中的 libbreakpad-core.so 失敗。
feifeideMacBook-Pro:try feifei$ mac/dump_syms libbreakpad-core.so > test.txt
libbreakpad-core.so: file is neither a fat binary file nor a Mach-O object file
但是src/processor下的minidump_stackwalk 可以直接處理android 項目生成的dmp文件的.
feifeideMacBook-Pro:processor feifei$ pwd
/Users/feifei/Desktop/tmp/breakpad/src/processor
feifeideMacBook-Pro:processor feifei$ ls | grep minidump_stackwalk
minidump_stackwalk
minidump_stackwalk.cc
minidump_stackwalk.o
minidump_stackwalk_machine_readable_test
minidump_stackwalk_test
/Users/feifei/Desktop/tmp/breakpad/src/processor/minidump_stackwalk 094495a4-c61d-473e-8505f986-72daf425.dmp > crash.info
最終解決辦法:安裝了一個ubuntn 虛擬機，在虛擬機中對breadpad 進行編譯，搞定~
五、breadpad 解析奔潰堆棧的另一種玩法
1、minidump_stackwalk 直接將.dmp 文件解析成可讀的信息，但是缺少堆棧的解析(奔潰點的函數符號和行號)
Android studo安裝目錄自帶了一份minidump_stackwalk自帶的minidump_stackwalk可以在mac上直接運行 解析dmp文件
查找minidump_stackwalk位置
find / -name minidump_stackwalk
我mac上的目錄為
/Applications/Android Studio.app/Contents/bin/lldb/bin/minidump_stackwalk
也可以直接使用“四”步中mac 環境編譯生成的 minidump_stackwalk,位置如下:
breakpad/src/src/processor/minidump_stackwalk
minidump_stackwalk 解析minidump文件
/Users/feifei/Library/Android/sdk/lldb/3.1/bin/minidump_stackwalk 094495a4-c61d-473e-8505f986-72daf425.dmp > crash.info
crash.info 內容:
Operating system: Android
0.0.0 Linux 4.4.83 #2 SMP PREEMPT Sun Jan 12 10:48:20 CST 2020 aarch64
CPU: arm64
4 CPUs
GPU: UNKNOWN
Crash reason: SIGSEGV /SEGV_MAPERR
Crash address: 0x0
Process uptime: not available
Thread 32 (crashed)
0 libbreakpad-core.so + 0x28468
x0 = 0x0000000000000027 x1 = 0x00000078e6b82f30
x2 = 0x0000000000000005 x3 = 0x0000000000000003
x4 = 0x0000000040100401 x5 = 0xa800000040404000
x6 = 0x0000000000000000 x7 = 0x7f7f7f7f7f7f7f7f
x8 = 0x0000000000000000 x9 = 0x0000000000000001
x10 = 0x00000078e6b830c0 x11 = 0x0000000000000018
x12 = 0x000000000000000b x13 = 0xffffffffffffffff
x14 = 0xff00000000000000 x15 = 0xffffffffffffffff
x16 = 0x00000078e86ec608 x17 = 0x00000078e8670454
x18 = 0x00000078e6b8166c x19 = 0x00000078fa1e8800
x20 = 0x0000007902ae7f20 x21 = 0x00000078fa1e8800
x22 = 0x00000078e6b8393c x23 = 0x00000078e9018c31
x24 = 0x0000000000000000 x25 = 0x00000078e6b84588
x26 = 0x00000078fa1e88a0 x27 = 0x0000000000000000
x28 = 0x0000000000000000 fp = 0x00000078e6b83650
lr = 0x00000078e86704c8 sp = 0x00000078e6b83620
pc = 0x00000078e8670468
Found by: given as instruction pointer in context
1 libbreakpad-core.so + 0x284c4
fp = 0x00000078e6b83670 lr = 0x00000078e8670508
sp = 0x00000078e6b83660 pc = 0x00000078e86704c8
Found by: previous frame's frame pointer
2 libbreakpad-core.so + 0x28504
fp = 0x00000078e6b83690 lr = 0x00000078e8670534
sp = 0x00000078e6b83680 pc = 0x00000078e8670508
Found by: previous frame's frame pointer
3 libbreakpad-core.so + 0x28530
由此可知程序 奔潰在了libbreakpad-core.so 的相對偏移位置 0x28468的地址
調用的關系如下：
libbreakpad-core.so + 0x28468
libbreakpad-core.so + 0x284c4
libbreakpad-core.so + 0x28504
libbreakpad-core.so + 0x28530
2、利用addr2line 根據發生crash的so文件，以及偏移地址，得出產生carsh的方法、行數和調用堆棧關系。
aarch64-linux-android-addr2line 工具也是在android sdk 安裝目錄下自帶的，可以自行查找。
feifeideMacBook-Pro:~ feifei$ find / -name aarch64-linux-android-addr2line
我mac上aarch64-linux-android-addr2line的目錄為
/Users/feifei/Library/Android/sdk/ndk-bundle/toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64/bin/aarch64-linux-android-addr2line
針對上面得出的調用堆棧，解析libbreakpad-core.so 的 0x28468、0x284c4、0x28504、0x28530分別對應哪個函數符號
libbreakpad-core.so + 0x28468
libbreakpad-core.so + 0x284c4
libbreakpad-core.so + 0x28504
libbreakpad-core.so + 0x28530
feifeideMacBook-Pro:try feifei$ /Users/feifei/Library/Android/sdk/ndk-bundle/toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64/bin/aarch64-linux-android-addr2line -f -C -e libbreakpad-core.so 0x28468 0x284c4 0x28504 0x28530
testCrash1()
??:?
testCoffeCacher()
??:?
call_dangerous_function()
??:?
Java_com_sogou_translate_jni_BreakpadInit_go2crash
??:?
可以得出實際的方法調用棧為:
testCrash1()
testCoffeCacher()
call_dangerous_function()
Java_com_sogou_translate_jni_BreakpadInit_go2crash
arm-linux-androideabi-addr2line 使用方法介紹:
arm-linux-androideabi-addr2line -C -f -e ${SOPATH} ${Address}
-C -f //打印錯誤行數所在的函數名稱
-e //打印錯誤地址的對應路徑及行數
${SOPATH} //so庫路徑
${Address} //需要轉換的堆棧錯誤信息地址，可以添加多個，但是中間要用空格隔開
