背景

近期使用uwsgi啟動django服務，發(fā)現(xiàn)在stop/reload uwsgi時會出現(xiàn)hangs問題，具體日志表現(xiàn)為：

...gracefully killing workers...
Gracefully killing worker 2 (pid: 390129)...
Gracefully killing worker 1 (pid: 390128)...
Gracefully killing worker 4 (pid: 390131)...
Gracefully killing worker 3 (pid: 390130)...
Wed Jul 31 20:13:14 2019 - worker 1 (pid: 390128) is taking too much time to die...NO MERCY !!!
Wed Jul 31 20:13:14 2019 - worker 2 (pid: 390129) is taking too much time to die...NO MERCY !!!
Wed Jul 31 20:13:14 2019 - worker 3 (pid: 390130) is taking too much time to die...NO MERCY !!!
Wed Jul 31 20:13:14 2019 - worker 4 (pid: 390131) is taking too much time to die...NO MERCY !!!
...

在測試中發(fā)現(xiàn)無論用什么防線先uwsgi發(fā)送stop/reload信號，uwsgi都會進入一種假死狀態(tài)，即不接收請求，也不結束進程。

期初以為uwsgi進程是在處理未完成的web請求。后來發(fā)現(xiàn)在沒有任何請求時，uwsgi也會進入這種夯死狀態(tài)。

問題原因

首先說明問題的原因是因何而起。

出現(xiàn)這種夯死的問題是由于在uwsgi中使用了線程導致。

Destroying threads in the POSIX world is basically no-way :) The right approach is letting your thread return when the destroy procedure is triggered. You have a bunch of ways to do it, but personally i am way more brutal and i generally set --worker-reload-mercy to a couple of seconds :)

在POSIX中銷毀一個線程幾乎是不可能實現(xiàn)的工作。正確的做法是：當接收到銷毀信號后正確的讓線程返回。通常有很多方式來實現(xiàn)著一點，但是更推薦的方式是設置一個--worker-reload-mercy時間。

解決方案

• 盡量避免在uwsgi中使用線程，首先使用線程會產(chǎn)生GIL鎖問題，其次在使用線程時還要考慮如何讓其正確退出。
• 設置reload-mercy和worker-reload-mercy兩個參數(shù)。

--reload-mercy - used when reloading/stopping whole uWSGI instance
--worker-reload-mercy - used when reloading/stopping single worker

問題復現(xiàn)

這里創(chuàng)建了一個最簡單的django服務，并用uwsgi來啟動。

進程模式

首先，配置uwsgi為進程模式啟動，這里創(chuàng)建了5個進程。

uwsgi配置文件如下：

[uwsgi]
socket = 127.0.0.1:8000
chdir = /opt/code/laji_backend
env = DJANGO_SETTINGS_MODULE=laji_baclend.settings
module = laji_baclend.wsgi:application
pidfile = /tmp/project-master.pid
master = True
processes = 5
home = /opt/code/laji_backend/venv
daemonize = /opt/log/uwsgi/laji_backend.log

現(xiàn)在來reload uwsgi服務，并查看uwsgi的日志。

...gracefully killing workers...
Gracefully killing worker 1 (pid: 13055)...
Gracefully killing worker 2 (pid: 13056)...
Gracefully killing worker 3 (pid: 13057)...
Gracefully killing worker 4 (pid: 13058)...
Gracefully killing worker 5 (pid: 13059)...
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
worker 5 buried after 1 seconds
......

結論：在uwsgi使用進程模式時，reload uwsgi不會出現(xiàn)夯死的問題。

線程模式

前面以進程方式啟動uwsgi沒有出現(xiàn)夯死問題，那么現(xiàn)在就試一下以線程模式啟動wusgi。這里啟動了5個進程，每個進程中又包含了兩個線程。

uwsgi配置文件如下：

[uwsgi]
socket = 127.0.0.1:8000
chdir = /opt/code/laji_backend
env = DJANGO_SETTINGS_MODULE=laji_baclend.settings
module = laji_baclend.wsgi:application
pidfile = /tmp/project-master.pid
master = True
enable-threads = True
processes = 5
threads = 2
home = /opt/code/laji_backend/venv
daemonize = /opt/log/uwsgi/laji_backend.log

reload uwsgi服務，并觀察日志輸出。

...gracefully killing workers...
Gracefully killing worker 1 (pid: 14913)...
Gracefully killing worker 2 (pid: 14914)...
Gracefully killing worker 3 (pid: 14915)...
Gracefully killing worker 4 (pid: 14916)...
Gracefully killing worker 5 (pid: 14917)...
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
worker 5 buried after 1 seconds
......

結論：在uwsgi中使用線程模式也不會造成reload夯死的問題。

uwsgi app中使用線程

uwsgi的線程不會造成任何問題，那前文所指的線程究竟是什么？

現(xiàn)在uwsgi啟動腳本中創(chuàng)建一個線程，在這種情況下嘗試reload uwsgi并查看日志輸出。

"""
WSGI config for laji_baclend project.

It exposes the WSGI callable as a module-level variable named ``application``.

For more information on this file, see
https://docs.djangoproject.com/en/2.2/howto/deployment/wsgi/
"""

import os
import time
import threading

from django.core.wsgi import get_wsgi_application

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'laji_baclend.settings')

def loop():
    while True:
        time.sleep(10)

t = threading.Thread(target=loop, name="loop_thread")
t.start()

application = get_wsgi_application()

隨后以線程方式啟動uwsgi，并reload uwsgi。

...gracefully killing workers...
Gracefully killing worker 1 (pid: 24390)...
Gracefully killing worker 2 (pid: 24391)...
Gracefully killing worker 3 (pid: 24394)...
Gracefully killing worker 4 (pid: 24396)...
Gracefully killing worker 5 (pid: 24398)...
Thu Aug  1 16:44:48 2019 - worker 1 (pid: 24390) is taking too much time to die...NO MERCY !!!
Thu Aug  1 16:44:48 2019 - worker 2 (pid: 24391) is taking too much time to die...NO MERCY !!!
Thu Aug  1 16:44:48 2019 - worker 3 (pid: 24394) is taking too much time to die...NO MERCY !!!
Thu Aug  1 16:44:48 2019 - worker 4 (pid: 24396) is taking too much time to die...NO MERCY !!!
Thu Aug  1 16:44:48 2019 - worker 5 (pid: 24398) is taking too much time to die...NO MERCY !!!
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
......

結論：在uwsgi app中使用線程就導致reload夯死。

采用reload-mercy和worker-reload-mercy避免

若場景中非要在uwsgi app中使用線程，可以通過配置reload-mercy和worker-reload-mercy兩個參數(shù)避免夯死的問題。

uwsgi配置文件如下：

[uwsgi]
socket = 127.0.0.1:8000
chdir = /opt/code/laji_backend
env = DJANGO_SETTINGS_MODULE=laji_baclend.settings
module = laji_baclend.wsgi:application
pidfile = /tmp/project-master.pid
master = True
enable-threads = True
processes = 5
threads = 2
home = /opt/code/laji_backend/venv
daemonize = /opt/log/uwsgi/laji_backend.log
reload-mercy = 1
worker-reload-mercy = 1

再次reload uwsgi服務，輸出日志如下：

...gracefully killing workers...
Gracefully killing worker 1 (pid: 24766)...
Gracefully killing worker 2 (pid: 24767)...
Gracefully killing worker 3 (pid: 24768)...
Gracefully killing worker 4 (pid: 24769)...
Gracefully killing worker 5 (pid: 24770)...
Thu Aug  1 16:50:13 2019 - worker 1 (pid: 24766) is taking too much time to die...NO MERCY !!!
Thu Aug  1 16:50:13 2019 - worker 2 (pid: 24767) is taking too much time to die...NO MERCY !!!
Thu Aug  1 16:50:13 2019 - worker 3 (pid: 24768) is taking too much time to die...NO MERCY !!!
Thu Aug  1 16:50:13 2019 - worker 4 (pid: 24769) is taking too much time to die...NO MERCY !!!
Thu Aug  1 16:50:13 2019 - worker 5 (pid: 24770) is taking too much time to die...NO MERCY !!!
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
worker 5 buried after 1 seconds
......

這里雖然也會出現(xiàn)NO MERCY問題，但是uwsgi在reload過程中并沒有出現(xiàn)夯死的情況。