ANR問題，相信是每位開發(fā)日常都會遇到的問題，對于這類問題的分析，按照官方的推薦，或網(wǎng)絡(luò)博客的總結(jié)思路能解決一定的問題，但是多數(shù)時候大家的困惑就是應(yīng)用本邏輯很簡單，耗時很短或應(yīng)用堆棧完全正常，或者或處于空閑狀態(tài)，可系統(tǒng)為什么就認為接收者發(fā)生ANR了呢？下面我將用幾個實例從不同角度分析導(dǎo)致ANR產(chǎn)生的Root Case。也希望對大家以后分析該類問題有一定參考價值。
ANR分類，含如下幾種類型：
廣播ANR
Service ANR
ContentProvider ANR
Input ANR
面向系統(tǒng)：WatchDog
產(chǎn)生ANR原因，如下幾種：
耗時操作
自身服務(wù)阻塞
系統(tǒng)阻塞
內(nèi)存緊張
CPU資源搶占
對于這些ANR，給大家的推薦一下大致分析思路和相關(guān)日志，通常發(fā)生ANR時，首先去查找對應(yīng)Trace日志，看看主線程是否在處理該廣播或被阻塞，如果發(fā)現(xiàn)上述現(xiàn)象，那么恭喜你，已經(jīng)很接近答案了。但如果發(fā)現(xiàn)堆棧完全處于空閑狀態(tài)，那么很不幸，就需要擴大參考面了，需要結(jié)合log日志進行分析，日志包括logcat, kernel日志，cpuinfo以及meminfo等，參考順序從前向后。
1分析logcat思路：首先在日志中搜索（“anr in”，“l(fā)ow_memory”, “slow_operation”）等關(guān)鍵字，通過該類關(guān)鍵字主要是查看系統(tǒng)Cpu負載，如果是發(fā)現(xiàn)應(yīng)用進程CPU明顯過高，那么很有可能是該進程搶占CPU過多導(dǎo)致，系統(tǒng)調(diào)度不及時,誤認為應(yīng)用發(fā)生了超時行為。
2分析kernel思路：在此類日志中直接搜索lowmemorykiller, 如果存在則查看發(fā)生時間和ANR時間是否大致對應(yīng)，相差無幾的話，可以從該日志中看到操作系統(tǒng)層面當(dāng)前內(nèi)存情況，F(xiàn)ree Memory說明的是空閑物理內(nèi)存，F(xiàn)ile Free說明的則是文件Cache，也就是應(yīng)用或系統(tǒng)從硬盤讀取文件，使用結(jié)束后，kernel并沒有這正釋放這類內(nèi)存，加以緩存，目的是為了下次讀寫過程加快速度。當(dāng)然，發(fā)現(xiàn)Free和Other整體數(shù)值都偏低時，Kernel會進行一定程度的內(nèi)存交換，導(dǎo)致整個系統(tǒng)卡頓。同時這類現(xiàn)象也會體現(xiàn)在log日志“slow_operation”中，即系統(tǒng)進程的調(diào)度也會收到影響。
3分析cpuinfo思路：這類日志一目了然，可以清晰的看到哪類進程CPU偏高，如果存在明顯偏高進程，那么ANR和此進程搶占CPU有一定關(guān)系。當(dāng)然，如發(fā)現(xiàn)Kswapd，emmc進程在top中，則說明遇到系統(tǒng)內(nèi)存壓力或文件IO開銷。
4分析meminfo思路：分析該類日志，主要是看哪類應(yīng)用或系統(tǒng)占用內(nèi)存偏高，如果應(yīng)用內(nèi)存占用比較正常，系統(tǒng)也沒有發(fā)生過度內(nèi)存使用，那么則說明系統(tǒng)中緩存了大量進程，并沒有及時釋放導(dǎo)致系統(tǒng)整體內(nèi)存偏低。
上面說了這么多，下面結(jié)合實例進行分析：
實例一：主線程進行耗時操作，或被進程內(nèi)其它線程阻塞
第一步，觀察Trace 主線程堆棧，發(fā)現(xiàn)主線程在申請內(nèi)存過程中被block，等待GC結(jié)束，但通過堆棧進一步發(fā)現(xiàn)其GC并沒有發(fā)生在該線程，也就是說在其他線程在執(zhí)行GC動作，而主線程在申請內(nèi)存過程中需要等待GC完成，再進一步申請內(nèi)存。
"main" prio=5 tid=1 WaitingForGcToComplete
native: #00 pc 0000000000019980? /system/lib64/libc.so(syscall+28)
native: #01 pc 000000000013a62c? /system/lib64/libart.so(_ZN3art17ConditionVariable4WaitEPNS_6ThreadE+136)
native: #02 pc 0000000000237f14? /system/lib64/libart.so(_ZN3art2gc4Heap19WaitForGcToCompleteENS0_7GcCauseEPNS_6ThreadE+1376)
native: #03 pc 000000000024798c? /system/lib64/libart.so(_ZN3art2gc4Heap22AllocateInternalWithGcEPNS_6ThreadENS0_13AllocatorTypeEmPmS5_S5_PPNS_6mirror5ClassE+168)
native: #04 pc 000000000050394c? /system/lib64/libart.so(artAllocObjectFromCodeRosAlloc+1412)
native: #05 pc 00000000001215d0? /system/lib64/libart.so(art_quick_alloc_object_rosalloc+64)
native: #06 pc 00000000018e72f0? /system/framework/arm64/boot.oat (Java_android_widget_TextView__0003cinit_0003e__Landroid_content_Context_2Landroid_util_AttributeSet_2II+1156)
at android.widget.TextView.(TextView.java:727)
at android.widget.TextView.(TextView.java:682)
at android.widget.TextView.(TextView.java:678)
at java.lang.reflect.Constructor.newInstance!(Native method)
第二步，再看看其它線程狀態(tài)，進一步查找發(fā)現(xiàn)，下面任務(wù)正在執(zhí)行GC
"LeuiRunningState:Background" prio=5 tid=28 WaitingPerformingGc
"AsyncTask #6" prio=5 tid=20 WaitingPerformingGc
綜上可以得出大致結(jié)論，Tid=28,20線程執(zhí)行GC,導(dǎo)致主線程申請內(nèi)存被Block.? 但是進一步思考，應(yīng)用GC是常有的事，但是為何這次需要這么長時間呢，帶著疑問我們看看進程的內(nèi)存使用情況：
Total number of allocations 9887486
Total bytes allocated 732MB
Total bytes freed 476MB
Free memory 5KB
Free memory until GC 5KB
Free memory until OOME 5KB
Total memory 256MB
Max memory 256MB
上面發(fā)現(xiàn)，應(yīng)用已使用256Mb, 距離OOM只有5K，內(nèi)存對象超過998萬個，也就是說GC過程需要掃描這些對象的巨大部分，導(dǎo)致耗時很久，另外內(nèi)存距離OOM只有5kb，說明有內(nèi)存泄漏，或內(nèi)存使用不合理。
綜上，對于這個問題得出結(jié)論，應(yīng)用進程內(nèi)存存在泄漏或使用不當(dāng)，導(dǎo)致GC時間過程，產(chǎn)生ANR.
實例二：應(yīng)用內(nèi)部線程邏輯依賴關(guān)系導(dǎo)致超時，觸發(fā)ANR
第一步，觀察Trace 主線程堆棧，發(fā)現(xiàn)主線程在Binder通信過程被Block.
"main" prio=5 tid=1 Native
| group="main" sCount=1 dsCount=0 obj=0x75f0eaa8 self=0x7fad046a00
| sysTid=4298 nice=-6 cgrp=default sched=0/0 handle=0x7fb1d18fe8
| state=S schedstat=( 79488910537 19985244611 169915 ) utm=6564 stm=1384 core=0 HZ=100
| stack=0x7fc237c000-0x7fc237e000 stackSize=8MB
| held mutexes=
kernel: (couldn't read /proc/self/task/4298/stack)
native: #00 pc 00000000000683d0? /system/lib64/libc.so(__ioctl+4)
native: #01 pc 00000000000723f8? /system/lib64/libc.so(ioctl+100)
native: #02 pc 000000000002d584? /system/lib64/libbinder.so(_ZN7android14IPCThreadState14talkWithDriverEb+164)
native: #03 pc 000000000002e050? /system/lib64/libbinder.so(_ZN7android14IPCThreadState15waitForResponseEPNS_6ParcelEPi+104)
native: #04 pc 000000000002e2c4? /system/lib64/libbinder.so(_ZN7android14IPCThreadState8transactEijRKNS_6ParcelEPS1_j+176)
native: #05 pc 0000000000025654? /system/lib64/libbinder.so(_ZN7android8BpBinder8transactEjRKNS_6ParcelEPS1_j+64)
native: #06 pc 00000000000e0928? /system/lib64/libandroid_runtime.so(???)
native: #07 pc 000000000139ba24? /system/framework/arm64/boot.oat (Java_android_os_BinderProxy_transactNative__ILandroid_os_Parcel_2Landroid_os_Parcel_2I+200)
at android.os.BinderProxy.transactNative(Native method)
at android.os.BinderProxy.transact(Binder.java:503)
at android.nfc.INfcAdapter$Stub$Proxy.setAppCallback(INfcAdapter.java:529)
at android.nfc.NfcActivityManager.requestNfcServiceCallback(NfcActivityManager.java:339)
at android.nfc.NfcActivityManager.setNdefPushMessageCallback(NfcActivityManager.java:309)
第二步，進一步查找此線程在和哪個進程進行通信，搜索關(guān)鍵字“setAppCallback”（Android命名習(xí)慣，客戶端和服務(wù)端函數(shù)命名基本相同），在Nfc的Binder_3線程響應(yīng)了客戶端請求，但在處理過程中被線程1阻塞，順著再看看線程1狀態(tài)
"Binder_3" prio=5 tid=17 Blocked
| group="main" sCount=1 dsCount=0 obj=0x12ddf0a0 self=0x7fa670f000
| sysTid=3183 nice=-6 cgrp=default sched=0/0 handle=0x7f93c30440
| state=S schedstat=( 3041465858 2637156615 16961 ) utm=168 stm=136 core=3 HZ=100
| stack=0x7f93b34000-0x7f93b36000 stackSize=1013KB
| held mutexes=
at com.android.nfc.P2pLinkManager.setNdefCallback(P2pLinkManager.java:420)
- waiting to lock <0x0bed0520> (a com.android.nfc.P2pLinkManager) held by thread 1
at com.android.nfc.NfcService$NfcAdapterService.setAppCallback(NfcService.java:1679)
at android.nfc.INfcAdapter$Stub.onTransact(INfcAdapter.java:178)
at android.os.Binder.execTransact(Binder.java:453)
"main" prio=5 tid=1 Native
| group="main" sCount=1 dsCount=0 obj=0x75f0eaa8 self=0x7fad046a00
| sysTid=2706 nice=0 cgrp=default sched=0/0 handle=0x7fb1d18fe8
| state=S schedstat=( 115355173189 36125520701 224819 ) utm=8594 stm=2941 core=0 HZ=100
| stack=0x7fc237c000-0x7fc237e000 stackSize=8MB
| held mutexes=
kernel: (couldn't read /proc/self/task/2706/stack)
native: #00 pc 00000000000683d0? /system/lib64/libc.so(__ioctl+4)
native: #01 pc 00000000000723f8? /system/lib64/libc.so(ioctl+100)
native: #02 pc 000000000002d584? /system/lib64/libbinder.so(_ZN7android14IPCThreadState14talkWithDriverEb+164)
native: #03 pc 000000000002e050? /system/lib64/libbinder.so(_ZN7android14IPCThreadState15waitForResponseEPNS_6ParcelEPi+104)
native: #04 pc 000000000002e2c4? /system/lib64/libbinder.so(_ZN7android14IPCThreadState8transactEijRKNS_6ParcelEPS1_j+176)
native: #05 pc 0000000000025654? /system/lib64/libbinder.so(_ZN7android8BpBinder8transactEjRKNS_6ParcelEPS1_j+64)
native: #06 pc 00000000000e0928? /system/lib64/libandroid_runtime.so(???)
native: #07 pc 000000000139ba24? /system/framework/arm64/boot.oat (Java_android_os_BinderProxy_transactNative__ILandroid_os_Parcel_2Landroid_os_Parcel_2I+200)
at android.os.BinderProxy.transactNative(Native method)
at android.os.BinderProxy.transact(Binder.java:503)
at android.nfc.IAppCallback$Stub$Proxy.createBeamShareData(IAppCallback.java:113)
at com.android.nfc.P2pLinkManager.prepareMessageToSend(P2pLinkManager.java:558)
- locked <0x0bed0520> (a com.android.nfc.P2pLinkManager)
通過主線程，又發(fā)現(xiàn)正進程Binder通信，同時被block,搜索關(guān)鍵字“createBeamShareData”，發(fā)現(xiàn)又回到瀏覽器線程，Binder_6線程響應(yīng)此請求，同時也處于Waiting狀態(tài)
"Binder_6" prio=5 tid=12 Waiting
| group="main" sCount=1 dsCount=0 obj=0x12c13a00 self=0x7f52850e00
| sysTid=23857 nice=0 cgrp=default sched=0/0 handle=0x7f694ff440
| state=S schedstat=( 705897380 828401158 3677 ) utm=45 stm=25 core=1 HZ=100
| stack=0x7f69403000-0x7f69405000 stackSize=1013KB
| held mutexes=
at java.lang.Object.wait!(Native method)
- waiting on <0x08a80433> (a java.lang.Object)
at java.lang.Thread.parkFor$(Thread.java:1220)
- locked <0x08a80433> (a java.lang.Object)
at sun.misc.Unsafe.park(Unsafe.java:299)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:810)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:970)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1278)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:203)
at com.android.browser.NfcHandler.createNdefMessage(NfcHandler.java:92)
at android.nfc.NfcActivityManager.createBeamShareData(NfcActivityManager.java:377)
at android.nfc.IAppCallback$Stub.onTransact(IAppCallback.java:53)
at android.os.Binder.execTransact(Binder.java:453)
為什么Binder_6處于Waiting狀態(tài)？這就需要大家結(jié)合Read the Fuck Code的精神研究邏輯了，事后發(fā)現(xiàn)此線程的事件放在了主線程執(zhí)行，當(dāng)執(zhí)行完畢后接收通知，停止waiting.
至此，我們找到了一條完整的鏈路，（瀏覽器主線程---->NFC Binder_3---->NFC主線程---->瀏覽器 Binder_6---->瀏覽器主線程），大家到此看到了根本原因，死鎖！！！
綜上，對于這個問題得出結(jié)論，應(yīng)用通信過程中發(fā)生死鎖導(dǎo)致ANR，后面只需解鎖即可。
上面兩類問題，相對簡單，大家遇到時也多半能自行分析解決，下面兩類涉及到較多系統(tǒng)或其它因素，問題比較隱晦，但是按照一定分析思路，靜下來分析，多數(shù)時候還是能找到原因或給出優(yōu)化方案的。
實例三：系統(tǒng)內(nèi)存過低，kernel進行內(nèi)存交換過程會引起整個系統(tǒng)運行緩慢(卡頓)
第一步，觀察Trace 主線程堆棧，發(fā)現(xiàn)主線程處于Suspend狀態(tài)；發(fā)生此類問題一般是兩種情況，一種是進程自身過于繁忙，每次分配時間片都不夠用，調(diào)度器強制把它置換成休眠了，另一種是系統(tǒng)比較繁忙，低優(yōu)先級集成得不到時間片；帶著這樣的疑問，繼續(xù)看：
"main" prio=5 tid=1 Suspended
| group="main" sCount=1 dsCount=0 obj=0x745518a0 self=0x7f86254a00
| sysTid=21916 nice=0 cgrp=default sched=0/0 handle=0x7f8b30efc8
| state=S schedstat=( 311762801762 96254728754 409881 ) utm=25610 stm=5566 core=0 HZ=100
| stack=0x7fd023c000-0x7fd023e000 stackSize=8MB
| held mutexes=
at java.util.regex.Splitter.fastSplit(Splitter.java:73)
at java.lang.String.split(String.java:1410)
at java.lang.String.split(String.java:1392)
at android.content.res.theme.LeResourceHelper.getResName(LeResourceHelper.java:193)
at android.content.res.Resources.loadDrawable(Resources.java:2624)
at android.content.res.Resources.getDrawable(Resources.java:862)
at android.content.Context.getDrawable(Context.java:458)
at android.widget.ImageView.resolveUri(ImageView.java:813)
這個時候可以看看應(yīng)用邏輯是否會存在繁忙操作不停搶占時間片，另一方面可以看看對應(yīng)日志，通過logcat發(fā)現(xiàn)如下信息，
11-17 09:49:41.392? 1532? 1574 E ActivityManager: ANR in com.android.systemui
11-17 09:49:41.392? 1532? 1574 E ActivityManager: PID: 21916
11-17 09:49:41.392? 1532? 1574 E ActivityManager: Reason: Broadcast of Intent { act=android.intent.action.TIME_TICK flg=0x50000014 mCallingUid=1000 (has extras) }
11-17 09:49:41.392? 1532? 1574 E ActivityManager: Load: 22.72 / 20.06 / 15.54? /分別對應(yīng)1分鐘/5分鐘/15分鐘/
11-17 09:49:41.392? 1532? 1574 E ActivityManager: CPU usage from 3ms to 24033ms later:
11-17 09:49:41.392? 1532? 1574 E ActivityManager:?? 60% 134/kswapd0: 0% user + 60% kernel
11-17 09:49:41.392? 1532? 1574 E ActivityManager:?? 32% 1532/system_server: 7.4% user + 25% kernel / faults: 31214 minor 423 major
系統(tǒng)整體負載很重，常規(guī)下負載在10左右；另外發(fā)現(xiàn)kswapdCPU占用率極高，通過這兩項可以得到系統(tǒng)內(nèi)存偏低，不停kill進程并發(fā)生內(nèi)存交換，是不是這樣的呢？我們再搜索一下其它關(guān)鍵字Slow operation：
11-17 09:42:25.292? 1532? 1572 W ActivityManager: Slow operation: 2440ms so far, now at startProcess: returned from zygote!
11-17 09:42:25.357? 1532? 1572 W ActivityManager: Slow operation: 2505ms so far, now at startProcess: done updating battery stats
11-17 09:42:25.357? 1532? 1572 I am_proc_start: [0,30188,10088,com.letv.android.usagestats,service,com.letv.android.usagestats/.UsageStatsReportService]
11-17 09:42:25.357? 1532? 1572 W ActivityManager: Slow operation: 2505ms so far, now at startProcess: building log message
11-17 09:42:25.357? 1532? 1572 I ActivityManager: Start proc 30188:com.letv.android.usagestats/u0a88 for service com.letv.android.usagestats/.UsageStatsReportService
11-17 09:42:25.357? 1532? 1572 W ActivityManager: Slow operation: 2505ms so far, now at startProcess: starting to update pids map
11-17 09:42:25.357? 1532? 1572 W ActivityManager: Slow operation: 2505ms so far, now at startProcess: done updating pids map
11-17 09:42:25.385? 1532? 1572 W ActivityManager: Slow operation: 2534ms so far, now at startProcess: done starting proc!
發(fā)現(xiàn)普通系統(tǒng)函數(shù)執(zhí)行一次就耗費了2S以上，足見系統(tǒng)卡頓。現(xiàn)在我們繼續(xù)延著內(nèi)存方向確認，看看meminfo日志吧
Total PSS by process:
3441530 kB: com.android.mms (pid 2518 / activities)
229272 kB: mediaserver (pid 763)
通過PSS發(fā)現(xiàn)，SMS進程內(nèi)存占用超過3G！對，第一反應(yīng)就是內(nèi)存泄漏，普通應(yīng)用甚至系統(tǒng)內(nèi)存占用根本不可能達到這么多。如果大家有時間可以看看kernel日志，搜索lowmemoryKiller，發(fā)生問題時間內(nèi)一定有大量的進程被kill.
綜上，對于這個問題得出結(jié)論，應(yīng)用在Native層發(fā)生內(nèi)存泄漏(不要問我為什么不是Java層發(fā)生這么多內(nèi)存泄漏@@)。導(dǎo)致系統(tǒng)整體內(nèi)存吃緊，又因為其本身Persist屬性，具有很高優(yōu)先級（-12），LMK不會將其Kill.只能不停Kill其它應(yīng)用，并進程內(nèi)存交換，類似問題參見XIIIM-8358
實例四：Binder資源耗盡，導(dǎo)致通信請求難以及時響應(yīng)
該類問題和內(nèi)存過低相似，查看主線程堆棧基本正常
"main" prio=5 tid=1 Native
| group="main" sCount=1 dsCount=0 obj=0x76261710 self=0x7f82646a00
| sysTid=3084 nice=0 cgrp=default sched=0/0 handle=0x7f874adfe8
| state=S schedstat=( 83808100322 29188718104 264083 ) utm=5716 stm=2664 core=1 HZ=100
| stack=0x7ff0f87000-0x7ff0f89000 stackSize=8MB
| held mutexes=
kernel: (couldn't read /proc/self/task/3084/stack)
native: #00 pc 00000000000682e4? /system/lib64/libc.so(__epoll_pwait+8)
native: #01 pc 000000000001f3a4? /system/lib64/libc.so(epoll_pwait+32)
native: #02 pc 000000000001be88? /system/lib64/libutils.so(_ZN7android6Looper9pollInnerEi+144)
native: #03 pc 000000000001c268? /system/lib64/libutils.so(_ZN7android6Looper8pollOnceEiPiS1_PPv+80)
native: #04 pc 00000000000d3088? /system/lib64/libandroid_runtime.so(_ZN7android18NativeMessageQueue8pollOnceEP7_JNIEnvP8_jobjecti+48)
native: #05 pc 000000000000554c? /system/framework/arm64/boot.oat (Java_android_os_MessageQueue_nativePollOnce__JI+144)
at android.os.MessageQueue.nativePollOnce(Native method)
at android.os.MessageQueue.next(MessageQueue.java:324)
at android.os.Looper.loop(Looper.java:135)
當(dāng)處于這種狀態(tài)時，我們直奔主題，分析log日志，按照logcat, kernel, cpuinfo, meminfo等依次分析：
11-08 23:51:44.088? 1514? 1554 E ActivityManager: ANR in com.android.phone
11-08 23:51:44.088? 1514? 1554 E ActivityManager: PID: 3084
11-08 23:51:44.088? 1514? 1554 E ActivityManager: Reason: Broadcast of Intent { act=com.android.internal.telephony.data-restart-trysetup.default flg=0x10000014 mCallingUid=1001 (has extras) }
11-08 23:51:44.088? 1514? 1554 E ActivityManager: Load: 9.92 / 9.81 / 10.02
11-08 23:51:44.088? 1514? 1554 E ActivityManager: CPU usage from 0ms to 6497ms later:
11-08 23:51:44.088? 1514? 1554 E ActivityManager:?? 108% 3084/com.android.phone: 101% user + 6.7% kernel / faults: 12120 minor 179 major
11-08 23:51:44.088? 1514? 1554 E ActivityManager:?? 66% 1514/system_server: 16% user + 49% kernel / faults: 20836 minor 88 major
11-08 23:51:44.088? 1514? 1554 E ActivityManager:?? 13% 13013/ca.bellmedia.cp24: 5.3% user + 8.4% kernel / faults: 3216 minor 39 major
通過上面的log日志，發(fā)現(xiàn)發(fā)生ANR進程本身CPU占用比較高，再搜索"slow operation"，“l(fā)ow_memory” 等關(guān)鍵字，都沒有出現(xiàn)在log日志中，而lowmemorykiller也以較合理的頻率出現(xiàn)在dmesg日志中，所以基本排除是內(nèi)存過低導(dǎo)致；所以下面延著CPU方向繼續(xù)分析
log日志無法找到更多線索，同時思考既然主線程狀態(tài)正常，那么高cpu一定是其它線程引起的，那就反饋trace繼續(xù)分析，查看phone進程的其它線程發(fā)現(xiàn)，幾乎所有binder線程都處于waiting狀態(tài)，只有Binder_2在工作：
"Binder_1" prio=5 tid=40 TimedWaiting
"Binder_3" prio=5 tid=40 TimedWaiting
"Binder_4" prio=5 tid=40 TimedWaiting
"Binder_5" prio=5 tid=39 TimedWaiting
"Binder_6" prio=5 tid=40 TimedWaiting
"Binder_7" prio=5 tid=40 TimedWaiting
"Binder_8" prio=5 tid=40 TimedWaiting
。。。。
"Binder_2" prio=5 tid=8 Native
| group="main" sCount=1 dsCount=0 obj=0x12c9b0a0 self=0x7f7be14400
| sysTid=3107 nice=0 cgrp=default sched=0/0 handle=0x7f8131d440
| state=R schedstat=( 515275891171 40426859698 234033 ) utm=49200 stm=2327 core=2 HZ=100
| stack=0x7f81221000-0x7f81223000 stackSize=1013KB
| held mutexes=
kernel: (couldn't read /proc/self/task/3107/stack)
native: #00 pc 0000000000070f20? /system/lib64/libsqlite.so(???)
native: #01 pc 000000000007420c? /system/lib64/libsqlite.so(sqlite3_step+652)
native: #02 pc 00000000000ba4a4? /system/lib64/libandroid_runtime.so(???)
native: #03 pc 00000000000ba514? /system/lib64/libandroid_runtime.so(???)
native: #04 pc 00000000003bc578? /system/framework/arm64/boot.oat (Java_android_database_sqlite_SQLiteConnection_nativeExecuteForChangedRowCount__JJ+140)
at android.database.sqlite.SQLiteConnection.nativeExecuteForChangedRowCount(Native method)
at android.database.sqlite.SQLiteConnection.executeForChangedRowCount(SQLiteConnection.java:732)
at android.database.sqlite.SQLiteSession.executeForChangedRowCount(SQLiteSession.java:754)
at android.database.sqlite.SQLiteStatement.executeUpdateDelete(SQLiteStatement.java:64)
at android.database.sqlite.SQLiteDatabase.delete(SQLiteDatabase.java:1499)
at com.android.providers.telephony.SmsProvider.delete(SmsProvider.java:899)
at android.content.ContentProvider$Transport.delete(ContentProvider.java:339)
at android.content.ContentProviderNative.onTransact(ContentProviderNative.java:206)
at android.os.Binder.execTransact(Binder.java:453)
進一步分析該線程狀態(tài)：state=R 說明其處于工作態(tài)。通過查看線程堆棧邏輯，發(fā)現(xiàn)正常情況下有l(wèi)og打印，借此再次返回到log日志，發(fā)現(xiàn)如下信息：
11-08 23:51:14.512? 3084? 3289 W SQLiteConnectionPool: The connection pool for database '/data/user/0/com.android.providers.telephony/databases/mmssms.db' has been unable to grant a connection to thread 111 (Binder_3) with flags 0x1 for 30.000002 seconds.
11-08 23:51:14.512? 3084? 3289 W SQLiteConnectionPool: Connections: 1 active, 0 idle, 0 available.
11-08 23:51:14.512? 3084? 3289 W SQLiteConnectionPool:
11-08 23:51:14.512? 3084? 3289 W SQLiteConnectionPool: Requests in progress:
11-08 23:51:14.512? 3084? 3289 W SQLiteConnectionPool:?? executeForChangedRowCount started 30008ms ago - running, sql="DELETE FROM sms WHERE (thread_id=2) AND (locked=0 AND date<1452658564000)"
11-08 23:51:14.513? 3084? 3613 W SQLiteConnectionPool: The connection pool for database '/data/user/0/com.android.providers.telephony/databases/mmssms.db' has been unable to grant a connection to thread 141 (Binder_5) with flags 0x1 for 30.009 seconds.
11-08 23:51:14.513? 3084? 3613 W SQLiteConnectionPool: Connections: 1 active, 0 idle, 0 available.
說明在Binder_3和Binder_6線程執(zhí)行Sql之前，已經(jīng)有其它線程執(zhí)行時間超過30S仍未結(jié)束。繼續(xù)搜集log發(fā)現(xiàn)，有15個Binder線程處于Waiting狀態(tài)，而那個正在執(zhí)行的則為Binder-2，耗時30S以上。
綜上了該進程高CPU的原因：Binder_2線程執(zhí)行Sql操作時間過長，進一步引起其它所有Binder線程被block，導(dǎo)致系統(tǒng)廣播發(fā)送無法及時通過Binder傳遞給主線程，誤觸發(fā)系統(tǒng)認為Phone進程處理廣播超時。
實例五：高CPU過度搶占時間片，導(dǎo)致其它應(yīng)用或任務(wù)難以及時調(diào)度
該類問題主線程多半是處于空閑或Suspend狀態(tài)，后者表示系統(tǒng)分配的CPU時間片無法滿足當(dāng)前需要便被強行切換，而引起該類現(xiàn)象的要么是底層系統(tǒng)動作，要么是其它任務(wù)高優(yōu)先級任務(wù)搶占CPU行為；
"main" prio=5 tid=1 Suspended
| group="main" sCount=2 dsCount=0 obj=0x75285af8 self=0x7f87a46a00
| sysTid=9251 nice=-6 cgrp=default sched=0/0 handle=0x7f8c5f7fe8
| state=S schedstat=( 50580737351 8433337317 81975 ) utm=4561 stm=497 core=1 HZ=100
| stack=0x7ff8105000-0x7ff8107000 stackSize=8MB
| held mutexes=
at java.util.Arrays.checkOffsetAndCount(Arrays.java:1722)
at java.nio.CharBuffer.wrap(CharBuffer.java:90)
at java.nio.CharBuffer.wrap(CharBuffer.java:68)
at android.text.TextDirectionHeuristics$TextDirectionHeuristicImpl.isRtl(TextDirectionHeuristics.java:149)
at android.text.BoringLayout.isBoring(BoringLayout.java:477)
at android.widget.TextView.onMeasure(TextView.java:7096)
at android.view.View.measure(View.java:19138)
at android.view.ViewGroup.measureChildWithMargins(ViewGroup.java:6064)
at android.widget.LinearLayout.measureChildBeforeLayout(LinearLayout.java:1465)
at android.widget.LinearLayout.measureHorizontal(LinearLayout.java:1112)
at android.widget.LinearLayout.onMeasure(LinearLayout.java:632)
at android.view.View.measure(View.java:19138)
當(dāng)Trace上無法繼續(xù)分析時，便需要分析日志了，搜索關(guān)鍵字“anr in”，發(fā)現(xiàn)
11-26 11:47:16.514? 1457? 1490 E ActivityManager: ANR in com.android.browser (com.android.browser/.MainActivity)
11-26 11:47:16.514? 1457? 1490 E ActivityManager: PID: 9251
11-26 11:47:16.514? 1457? 1490 E ActivityManager: Reason: Input dispatching timed out (Waiting to send non-key event because the touched window has not finished processing certain input events that were delivered to it over 500.0ms ago.? Wait queue length: 10.? Wait queue head age: 8974.9ms.)
11-26 11:47:16.514? 1457? 1490 E ActivityManager: Load: 10.97 / 10.71 / 10.0
11-26 11:47:16.514? 1457? 1490 E ActivityManager: CPU usage from 0ms to 10480ms later:
11-26 11:47:16.514? 1457? 1490 E ActivityManager:?? 114% 9251/com.android.browser: 65% user + 48% kernel / faults: 10870 minor 11 major
11-26 11:47:16.514? 1457? 1490 E ActivityManager:?? 108% 1457/system_server: 33% user + 74% kernel / faults: 9584 minor 11 major
瀏覽器自身CPU占用較高，至于System_server占用比較多，尤其是當(dāng)大家看到“ CPU usage from 0ms to 10480ms later”已經(jīng)kernel部分（74% kernel /）占用較多的情況下，不要再輕易懷疑是system_server高CPU導(dǎo)致，其高CPU的真正原因是需要dump各進程信息而已。
順著"ANR in"之前的日志，我們繼續(xù)向上看，發(fā)現(xiàn)該應(yīng)該進行了大量且頻繁的GC操作
11-26 11:47:05.204? 1457? 1467 I art???? : Background partial concurrent mark sweep GC freed 842(578KB) AllocSpace objects, 455(85MB) LOS objects, 8% free, 169MB/185MB, paused 2.140ms total 245.072ms
11-26 11:47:10.493? 9251 31938 W art???? : Suspending all threads took: 131.446ms
11-26 11:47:10.598? 9251 31938 W art???? : Suspending all threads took: 88.134ms
11-26 11:47:10.699? 9251 31938 W art???? : Suspending all threads took: 93.939ms
11-26 11:47:10.795? 9251 31938 W art???? : Suspending all threads took: 75.051ms
11-26 11:47:10.821? 9251 31938 W art???? : Suspending all threads took: 14.536ms
11-26 11:47:10.956? 9251 31938 W art???? : Suspending all threads took: 114.243ms
11-26 11:47:11.101? 9251 31938 W art???? : Suspending all threads took: 121.775ms
11-26 11:47:11.254? 9251 31938 W art???? : Suspending all threads took: 93.763ms
.....
而根據(jù)GC類型（Background partial concurrent）來看，應(yīng)該是有任務(wù)在不停的申請和使用大量內(nèi)存，帶著這樣的想法，需要再此返回到Trace日志，分析相關(guān)線程狀態(tài)，在大量的對比分析篩選之后，很幸運的發(fā)現(xiàn)了如下線程（該線程只有采集TraceView才會出現(xiàn)），并且處于R狀態(tài)。對TraceView了解的同事都知道，該任務(wù)會引起關(guān)聯(lián)進程非常大CPU消耗，并且異常卡頓（主線程得不到及時響應(yīng)）。
"Sampling Profiler" daemon prio=9 tid=162 Native
| group="system" sCount=1 dsCount=0 obj=0x13102220 self=0x7f5a82f800
| sysTid=31938nice=-6cgrp=default sched=0/0 handle=0x7f643ff440
|state=Rschedstat=( 22112458218 4449717737 10001 ) utm=2021 stm=190 core=0 HZ=100
綜上找到了該進程高CPU的原因：采集TraceView線程需要申請大量內(nèi)存不斷觸發(fā)進程內(nèi)部GC，并且自身任務(wù)屬于高耗時操作，從未導(dǎo)致主線程得不到及時調(diào)度和響應(yīng)，觸發(fā)ANR。
實例六：日志不全，缺少Trace或其它日志
遇到這類問題是比較郁悶的，這個時候智能拿現(xiàn)有的信息進行分析，嘗試找出問題或改進方向，例如缺少Trace.但是其它日志相對齊全
例如在event日志中找到了應(yīng)用ANR的大概時間點：10-14 00:40:26.010650
10-14 00:40:26.010650? 1132? 1172 I am_anr? : [0,19746,android.process.media,952680005,Broadcast of Intent { act=android.intent.action.MEDIA_SCANNER_SCAN_FILE dat=file:///sdcard/AutoSmoke_UI30/testSwitchLetvView_20161014_003533/1476376700108.png在flg=0x10 cmp=com.android.providers.media/.MediaScannerReceiver }]
在sys_log中發(fā)現(xiàn)ANR時進程CPU信息
10-14 00:40:57.052274? 1132? 1172 E ANRManager: ANR in android.process.media, time=304722739
10-14 00:40:57.052274? 1132? 1172 E ANRManager: Reason: Broadcast of Intent { act=android.intent.action.MEDIA_SCANNER_SCAN_FILE dat=file:///sdcard/AutoSmoke_UI30/testSwitchLetvView_20161014_003533/1476376700108.pngflg=0x10 cmp=com.android.providers.media/.MediaScannerReceiver }
10-14 00:40:57.052274? 1132? 1172 E ANRManager: Load: 37.88 / 25.54 / 20.22
10-14 00:40:57.052274? 1132? 1172 E ANRManager: Android time :[2016-10-14 00:40:56.95] [304754.500]
10-14 00:40:57.052274? 1132? 1172 E ANRManager: CPU usage from 17448ms to 0ms ago:
10-14 00:40:57.052274? 1132? 1172 E ANRManager:?? 117% 19252/com.letv.android.letvlive: 80% user + 36% kernel / faults: 684 minor
10-14 00:40:57.052274? 1132? 1172 E ANRManager:?? 110% 11620/mediaserver: 64% user + 45% kernel / faults: 23 minor
10-14 00:40:57.052274? 1132? 1172 E ANRManager:?? 41% 378/logd: 19% user + 21% kernel / faults: 17 minor
10-14 00:40:57.052274? 1132? 1172 E ANRManager:?? 22% 573/mobile_log_d: 17% user + 5.3% kernel / faults: 1123 minor
10-14 00:40:57.052274? 1132? 1172 E ANRManager:?? 18% 19286/com.letv.android.letvlive:cde: 11% user + 6.9% kernel / faults: 6029 minor
10-14 00:40:57.052274? 1132? 1172 E ANRManager:?? 18% 422/adbd: 2.1% user + 15% kernel / faults: 1722 minor
10-14 00:40:57.052274? 1132? 1172 E ANRManager:?? 17% 18392/logcat: 7.4% user + 10% kernel
從上面日志可以看到有兩個進程CPU占用率偏高，且系統(tǒng)長時間CPU負載很重（Load: 37.88 / 25.54 / 20.22），尤其是ANR之前1分鐘的負載達到37；由此我們可以大概率的猜測這次ANR事故是由CPU過高導(dǎo)致其它任務(wù)調(diào)度不及時導(dǎo)致，到底是不是呢？還是如其他同事認為的內(nèi)存原因引起呢？下面我們繼續(xù)看對應(yīng)時間點的Kernel日志，關(guān)鍵字”lowmemorykiller“，得到如下信息：
<6>[302600.931727]? (4)[10628:Cam@AuxSensorCo]lowmemorykiller: Killing 'android.browser' (28649), adj 18, score_adj 1000,
<6>[302600.931727]??? to free 72464kB on behalf of 'Cam@AuxSensorCo' (10628) because
<6>[302600.931727]cache 1000628kBis below limit 322560kB for oom_score_adj 0
<6>[302600.931727]??? Free memory is 235708kB above reserved
<6>[303901.663086]? (6)[16560:Cam@AuxSensorCo]lowmemorykiller: Killing 'roid.emojistore' (15854), adj 18, score_adj 1000,
<6>[303901.663086]??? to free 75636kB on behalf of 'Cam@AuxSensorCo' (16560) because
<6>[303901.663086]cache 1292884kBis below limit 322560kB for oom_score_adj 0
<6>[303901.663086]??? Free memory is 285336kB above reserved
<6>[302623.705248]? (2)[10970:Cam@AuxSensorCo]lowmemorykiller: Killing 'ews:pushservice' (6186), adj 13, score_adj 764,
<6>[302623.705248]??? to free 62140kB on behalf of 'Cam@AuxSensorCo' (10970) because
<6>[302623.705248]cache 992668kBis below limit 322560kB for oom_score_adj 0
<6>[302623.705248]??? Free memory is 81320kB above reserved
cache項 ：為kernel端的文件緩存cache，為了提高IO訪問速度，底層系統(tǒng)會有選擇的緩存一些文件；
limit：內(nèi)存（文件緩存）的最低內(nèi)存限制322560kB，當(dāng)內(nèi)存和文件緩存同時低于這個閥值，LMK變開始尋找低優(yōu)先級進程查殺。
score_adj：從上層設(shè)置到kernel經(jīng)過轉(zhuǎn)換后的進程優(yōu)先級，adj--> score_adj; score_adj為1000，則說明被查殺的進程優(yōu)先級很低。
Free memory：當(dāng)前空閑物理內(nèi)存。
[302623.705248]：Kernel開機時間戳
通過以上日志分析可以得出結(jié)論：系統(tǒng)可用內(nèi)存（Free+Cache）整體維持在1G左右，屬于良好。查殺進程間隔時間較長，不會對系統(tǒng)負載帶來太多開銷。
分析完以上日志，基本排除了內(nèi)存問題引起的ANR，接下來再回到log日志，分析ANR高CPU進程的相關(guān)日志，看看能否有進一步挖掘。在log日志中，高亮進程PID(11620)，結(jié)果發(fā)現(xiàn)在很長一段時間內(nèi)存，該進程有幾十萬的日志輸出，此時心里或許有了希望，這么頻繁的輸出，且含有很多相同日志，那就說明該進程產(chǎn)生了大量循環(huán)，而大量循環(huán)也是高CPU的常見起因。
10-14 00:40:46.0357071162019687 D MtkOmxVdecEx: [0xe1eb7800] RemoveInputBuf frm=0xe1eb8d70, omx=0xa3b9dfe0, i=5
10-14 00:40:46.0357911162019687 D MtkOmxVdecEx: [0xe1eb7800] FB in (0xA3B9DFE0)
10-14 00:40:46.036599 11620 11620 D MtkOmxMVAMgr: [0xb3cca9f0] [ION][FreeBuffer] entry=0xa3bcf3c0, va=0xd30d7000, pa=0x47600000,size=0x180000, srcFd=0xFFFFFFFF, fd=0xFFFFFFFF, bufHdr=0xA3B9CAE0
10-14 00:40:46.037036 11620 11620 D MtkOmxVdecEx: [0xe1eb7800] RemoveInputBuf frm=0xe1eb8d28, omx=0xa3b9cae0, i=4
10-14 00:40:46.037125 11620 11620 D MtkOmxVdecEx: [0xe1eb7800] FB in (0xA3B9CAE0)
10-14 00:40:46.037907 11620 11655 D MtkOmxMVAMgr: [0xb3cca9f0] [ION][FreeBuffer] entry=0xabbfc4e0, va=0xd3557000, pa=0x47000000,size=0x180000, srcFd=0xFFFFFFFF, fd=0xFFFFFFFF, bufHdr=0xA3B9C0C0
10-14 00:40:46.038281 11620 11655 D MtkOmxVdecEx: [0xe1eb7800] RemoveInputBuf frm=0xe1eb8ce0, omx=0xa3b9c0c0, i=3
10-14 00:40:46.038364 11620 11655 D MtkOmxVdecEx: [0xe1eb7800] FB in (0xA3B9C0C0)
10-14 00:40:46.039097 11620 11657 D MtkOmxMVAMgr: [0xb3cca9f0] [ION][FreeBuffer] entry=0xa3bcf240, va=0xd3f80000, pa=0x46c00000,size=0x180000, srcFd=0xFFFFFFFF, fd=0xFFFFFFFF, bufHdr=0xA3B9C120
10-14 00:40:46.039734 11620 11657 D MtkOmxVdecEx: [0xe1eb7800] RemoveInputBuf frm=0xe1eb8c98, omx=0xa3b9c120, i=2
10-14 00:40:46.039829 11620 11657 D MtkOmxVdecEx: [0xe1eb7800] FB in (0xA3B9C120)
10-14 00:40:46.041510 11620 11653 D MtkOmxMVAMgr: [0xb3cca9f0] [ION][FreeBuffer] entry=0xa3bcf6f0, va=0xdb528000, pa=0x46600000,size=0x180000, srcFd=0xFFFFFFFF, fd=0xFFFFFFFF, bufHdr=0xA3B9DF20
10-14 00:40:46.041966 11620 11653 D MtkOmxVdecEx: [0xe1eb7800] RemoveInputBuf frm=0xe1eb8c50, omx=0xa3b9df20, i=1
10-14 00:40:46.042057 11620 11653 D MtkOmxVdecEx: [0xe1eb7800] FB in (0xA3B9DF20)
10-14 00:40:46.043345 11620 11654 D MtkOmxMVAMgr: [0xb3cca9f0] [ION][FreeBuffer] entry=0xa3bcf120, va=0xdb828000, pa=0x43200000,size=0x180000, srcFd=0xFFFFFFFF, fd=0xFFFFFFFF, bufHdr=0xABBC4420
10-14 00:40:46.043756 11620 11654 D MtkOmxVdecEx: [0xe1eb7800] RemoveInputBuf frm=0xe1eb8c08, omx=0xabbc4420, i=0
10-14 00:40:46.043841 11620 11654 D MtkOmxVdecEx: [0xe1eb7800] FB in (0xABBC4420)
10-14 00:40:46.044026 11620 11654 D MtkOmxVdecEx: [0xe1eb7800] MtkOmxVdec::FreeBuffer all input buffers have been freed!!! signal mInPortFreeDoneSem(1)
至此，導(dǎo)出進一步結(jié)論，應(yīng)用發(fā)生ANR主要是上面兩個進程高CPU引起調(diào)度不及時。至于進程高CPU的進一步原因，則需要相關(guān)模塊Owner結(jié)合日志進一步分析，論證。
通過以上6類ANR實例剖析，可以看出，除了正常Receiver處理耗時操作引起的ANR之外，也會有其它因素引發(fā)此類問題，例如總體內(nèi)存偏低導(dǎo)致交換（kswap），CPU過高導(dǎo)致調(diào)度不及時，Binder資源被耗盡無法及時通訊等等，發(fā)生此類問題線索較為隱晦，需要大家匯總多個日志反復(fù)對比；但是好在這類問題發(fā)生時，系統(tǒng)都有關(guān)鍵log日志輸出，可以利用關(guān)鍵字多角度深入分析，綜合對比，這類問題多數(shù)時候是可以得出有效結(jié)論，并給出優(yōu)化(解決)方案；對應(yīng)日志確實不足的，只能借助于測試幫忙復(fù)現(xiàn)，并提供更多有效日志了。除此之外，也需要對相關(guān)系統(tǒng)知識有更多了解，例如LMK, 進程調(diào)能，Binder通信機制。方能在分析，解決此類問題過程中，有更多的參考和衡量。
