歡迎到我的個人博客閱讀這篇文章,體驗更好哦~
需求
用 drf
寫接口,希望在請求成功后,能記錄一些信息,以便于追溯問題
- 將當前請求的
request
、response
數據記錄到本地log
文件中 - 將關鍵信息上報至
sentry
- 為了能更好的查詢日志,希望每個請求到響應的日志中,有一個全局唯一的
log-uuid
明確記錄的數據
首先來確定下我們需要哪些數據,以便于下一步的設計。
::: warning 注意
應該明白,我的目的僅僅是記錄請求-響應這個過程中的數據,而不包含記錄業務相關數據。
業務相關數據,還是要到業務代碼中進行記錄。
:::
- request
- uid,用戶 uuid,如果是未登錄或者認證失敗則返回 None
- ip,用戶的遠程 ip 地址
- path,api的地址,重要
- view,訪問的 view 視圖函數
- method,請求方式
- body,請求的參數,重要
- headers,請求的 headers,重要
- response
- status,http狀態碼
- body,響應的 body,重要
- headers,響應的 headers,重要
其次,我計劃將 log-uuid,放在響應的 headers 里面返回給客戶端,即 response.headers 里面。
那么,無論是記錄本地 log 還是上報 sentry,以上數據基本能滿足,不滿足的再自己根據實際情況添加即可。
如何記錄
View 視圖函數中記錄
由于要記錄每個請求的request 、response信息,我們首先可以想到 view 視圖。
APIView
如果你的視圖函數是 CBV,繼承自 APIView或者是使用 FBV 的話,可以在每個視圖函數中,log 記錄request和response,例如:
::: details 展開查看代碼
class TestView(APIView):
# CBV 方式
def post(self, request, *args, **kwargs):
log.info(request.data) # 記錄請求信息
resp = {
'msg': 'ok',
'code': 20000,
'data': None
}
log.info(resp) # 記錄響應信息
sentry_sdk.capture_message('something') # 上報 sentry
return APIResponse(resp) # APIResponse是自定義的響應類
:::
很顯然,有多少個視圖函數,就得都在里面加上這兩個 log.info(),以及上報 sentry 的代碼,那代碼太冗余,肯定是不科學的。
ModelViewSet
如果你的視圖函數是繼承自 DRF 自身的 ModelViewSet,好像就沒有辦法在視圖中記錄了
如果你自定義了一個 ModelViewSet,暫且稱為 APIModelViewSet,那么可以在 APIModelViewSet 類中實現日志記錄,例如:
::: details 展開查看代碼
from rest_framework.exceptions import ValidationError
from rest_framework.viewsets import ModelViewSet
from buggg.core.response import APIResponse
from buggg.utils.utils import now
class APIModelViewSet(ModelViewSet):
"""
自定義ViewSet,主要實現在寫數據到model時保存當前操作用戶,及返回response時使用自定義的APIResponse類、記錄日志
"""
def list(self, request, *args, **kwargs):
log.info('record request')
queryset = self.filter_queryset(self.get_queryset())
page = self.paginate_queryset(queryset)
if page is not None:
serializer = self.get_serializer(page, many=True, context={'request': self.request})
return self.get_paginated_response(serializer.data)
serializer = self.get_serializer(queryset, many=True, context={'request': self.request})
log.info('record response')
return APIResponse(serializer.data)
def retrieve(self, request, *args, **kwargs):
log.info('record request')
instance = self.get_object()
serializer = self.get_serializer(instance, context={'request': self.request})
log.info('record response')
return APIResponse(serializer.data)
def create(self, request, *args, **kwargs):
log.info('record request')
serializer = self.get_serializer(data=request.data, context={'request': self.request})
if serializer.is_valid():
model = serializer.Meta.model
if hasattr(model, 'create_by'):
serializer.save(create_by=request.user)
if hasattr(model, 'update_by'):
serializer.save(update_by=request.user)
if not hasattr(model, 'create_by') and not hasattr(model, 'update_by'):
serializer.save()
headers = self.get_success_headers(serializer.data)
log.info('record response')
return APIResponse(serializer.data, msg='創建成功', headers=headers)
raise ValidationError(serializer.errors)
def update(self, request, *args, **kwargs):
log.info('record request')
partial = kwargs.pop('partial', False)
instance = self.get_object()
serializer = self.get_serializer(instance, data=request.data, partial=partial,
context={'request': self.request})
if serializer.is_valid():
model = serializer.Meta.model
if hasattr(model, 'update_by'):
serializer.save(update_by=request.user)
else:
serializer.save()
log.info('record response')
return APIResponse(serializer.data, msg='更新成功')
raise ValidationError(serializer.errors)
def destroy(self, request, *args, **kwargs):
log.info('record request')
instance = self.get_object()
self.perform_destroy(instance)
if hasattr(instance, 'update_by'):
instance.update_by = request.user
if hasattr(instance, 'update_time'):
instance.update_time = now()
instance.save()
log.info('record response')
return APIResponse(msg='刪除成功')
:::
這種自定義ViewSet的方式,可以在方法處進行日志記錄,比在每個視圖函數里記錄要省很多事,明顯好多了。
但是,我們還有一個需求是 希望每個請求到響應的日志中,有一個全局唯一的 log-uuid,并且通過響應 headers 返回給客戶端
再繼續改造改造 APIResponse,在返回時加一個 header,好像也是能做到的。但是仔細分析下這種方法,并不通用,因為假如自己沒有繼承自 ModelViewSet,而是繼承自其他的視圖類,那又要重新去弄了,不合理。
Middleware中記錄
我們知道,當 django 注冊了多個中間件時,先從上至下執行 process_request方法,再從下至上執行 process_response 方法。
::: tip
不了解 django 中間件的可以去百度搜索一下,很多講解。
中間件主要有4個方法,本次我們主要接觸 process_request 和 process_response
:::
根據我們的需求,需要記錄日志,需要上報 sentry,需要生成一個全局的 log-uuid并且由 response.headers 返回給客戶端。那拆解下,把記錄日志作為一個中間件,把上報 sentry 作為一個中間件,互相隔離開,不在同一個中間件中完成。
假設日志記錄的中間件為LoggerMiddleware,上報 sentry 的中間件為SentryMiddleware,可以大概完成代碼如下
::: details 展開查看LoggerMiddleware代碼
import time
import sentry_sdk
from django.contrib.auth.models import AnonymousUser
from django.utils.deprecation import MiddlewareMixin
from buggg.utils.log import log
from buggg.utils.utils import gen_uuid
class LoggerMiddleware(MiddlewareMixin):
def process_request(self, request):
request.body = str(request.body, encoding='utf-8').replace(' ', '').replace('\n',
'').replace('\t',
'')
if 'HTTP_X_FORWARDED_FOR' in request.META:
remote_address = request.META['HTTP_X_FORWARDED_FOR']
else:
remote_address = request.META['REMOTE_ADDR']
request.remote_address = remote_address
# 此處省略一些數據提取步驟
log.info(request) # 記錄請求數據,注意此處是簡寫了,實際要對request數據進行一些處理
def process_response(self, request, response):
# 獲取響應內容
# 獲取響應內容
if response['content-type'] == 'application/json':
if getattr(response, 'streaming', False):
response_body = '<<<Streaming>>>'
else:
response_body = str(response.content, encoding='utf-8')
else:
response_body = '<<<Not JSON>>>'
request.response_body = response_body
# 此處省略一些數據提取步驟
log.info(response) # 記錄響應數據,注意此處是簡寫了,實際要對response數據進行一些處理
return response
class SentryMiddleware(MiddlewareMixin):
def process_request(self, request):
request.body = str(request.body, encoding='utf-8').replace(' ', '').replace('\n',
'').replace('\t',
'')
if 'HTTP_X_FORWARDED_FOR' in request.META:
remote_address = request.META['HTTP_X_FORWARDED_FOR']
else:
remote_address = request.META['REMOTE_ADDR']
request.remote_address = remote_address
sentry_sdk.add_breadcrumb(
category='remote_address',
message=f'{request.remote_address}',
level='info',
)
sentry_sdk.add_breadcrumb(
category='body',
message=request.request_body,
level='info',
)
def process_response(self, request, response):
return response
:::
通過這2個中間件的代碼,可以發現,process_request里面存在一些相同的代碼,而這些代碼的作用就是獲取 request 或者 response 信息的。既然這里存在相同的代碼,那能不能把這部分再提取下呢。
我們再回憶一下中間件的處理順序,先從上至下執行 process_request方法,再從下至上執行 process_response 方法,在執行的過程中,request 對象和 response 對象都是會一直傳遞的,傳遞的順序就和執行的方法順序一致。
假設我們定義這樣兩個中間件,第一個暫且稱為ReqCollectionMiddleware,作用是提取 request 信息保存起來且傳遞下去;第二個暫且稱為ResCollectionMiddleware,作用是提取 response 信息保存起來且傳遞下去。
ReqCollectionMiddleware是提取request,那么就要在process_request方法里面做文章了,我們可以在這個中間件的process_request里面把我們需要的 request 信息提取出來,保存到 request.META 中,而這個中間件的 process_response 則不需要做任何處理,實現代碼如下:
::: details 展開查看代碼
class ReqCollectionMiddleware(MiddlewareMixin):
def process_request(self, request):
log.info('進入ReqCollectionMiddleware,收集請求數據')
request.META['REQUEST_BODY'] = json.loads(str(request.body, encoding='utf-8').replace(' ', '').replace('\n',
'').replace(
'\t',
''))
if 'HTTP_X_FORWARDED_FOR' in request.META:
remote_address = request.META['HTTP_X_FORWARDED_FOR']
else:
remote_address = request.META['REMOTE_ADDR']
request.META['IP'] = remote_address
request.META['LOG_UUID'] = gen_uuid()
def process_response(self, request, response):
return response
:::
同時,在第18行,可以看到我傳遞一個 uuid 給 request.META['LOG_UUID']
,這將為后續做準備。
ResCollectionMiddleware是提取response,那么就要在 process_response 方法里面做文章了,我們可以在這個中間件的process_response里面把我們需要的 response 信息提取出來。由于我們在上面已經把需要的數據放進了request.META 中,所以我們索性將 response 的數據也保存到其中。而這個中間件的 process_request 則不需要做任何處理,實現代碼如下:
::: details 展開查看代碼
class ResCollectionMiddleware(MiddlewareMixin):
def process_request(self, request):
pass
def process_response(self, request, response):
log.info('進入ResCollectionMiddleware,收集響應數據')
# 獲取請求的 uid,如果是未登錄的則為 None
if not isinstance(request.user, AnonymousUser):
uid = request.user.uid
else:
uid = None
request.META['USER_UID'] = uid
# 獲取響應內容
if response['content-type'] == 'application/json':
if getattr(response, 'streaming', False):
response_body = '<<<Streaming>>>'
else:
response_body = json.loads(str(response.content, encoding='utf-8'))
else:
response_body = '<<<Not JSON>>>'
request.META['RESP_BODY'] = response_body
# 獲取請求的 view 視圖名稱
try:
request.META['VIEW'] = request.resolver_match.view_name
except AttributeError:
request.META['VIEW'] = None
request.META['STATUS_CODE'] = response.status_code
# 設置 headers: X-Log-Id
response.setdefault('X-Log-Id', request.META['LOG_UUID'])
return response
:::
同時,在第36行,可以看到我為 response 設置了一個 header,名為X-Log-Id
,值為我之前生成的 uuid
至此,通過這2個中間件,我們就完成了 request 和 response 信息提取,以及生成一個 log-uuid,并且還能在 response headers里面返回給客戶端,這樣有問題就好追溯了。那么接下來,LoggerMiddleware 和 SentryMiddleware 就可以簡化一些了。
::: details 展開查看LoggerMiddleware代碼
class LoggerMiddleware(MiddlewareMixin):
"""
中間件,記錄日志
"""
def process_request(self, request):
pass
def process_response(self, request, response):
log.info('進入LoggerMiddleware,寫日志')
log_data = {
'request': {
'uid': request.META['USER_UID'],
"ip": request.META['IP'],
"method": request.method,
'path': request.get_full_path(),
'view': request.META['VIEW'],
'body': request.META['REQUEST_BODY'],
'headers': _get_request_headers(request),
},
'response': {
'status': request.META['STATUS_CODE'],
'body': request.META['RESP_BODY'],
'headers': _get_response_headers(response),
},
'log_uuid': request.META['LOG_UUID']
}
log.info(json.dumps(log_data))
return response
:::
::: details 展開查看SentryMiddleware代碼
class SentryMiddleware(MiddlewareMixin):
def process_request(self, request):
pass
def process_response(self, request, response):
log.info('進入SentryMiddleware,上報請求至Sentry')
sentry_sdk.add_breadcrumb(
category='path',
message=request.path,
level='info',
)
sentry_sdk.add_breadcrumb(
category='body',
message=request.META["REQUEST_BODY"],
level='info',
)
sentry_sdk.add_breadcrumb(
category='request_headers',
message=_get_request_headers(request),
level='info',
)
sentry_sdk.add_breadcrumb(
category='response_headers',
message=_get_response_headers(response),
level='info',
)
sentry_sdk.add_breadcrumb(
category='view',
message=request.META['VIEW'],
level='info',
)
sentry_sdk.set_user({"id": request.META['USER_UID']})
sentry_sdk.set_tag("log-id", request.META["LOG_UUID"])
sentry_sdk.capture_message(request.META["LOG_UUID"])
return response
:::
::: tip 提示
由于沒有提供直接獲取 headers 的方法,所以
_get_response_headers() 是自定義的獲取完整 response headers 的方法
_get_request_headers() 是自定義的獲取完整 request headers 的方法
代碼在后面會給出
:::
再次優化
我發現,可以將ReqCollectionMiddleware和ResCollectionMiddleware合并為一個中間件CollectionMiddleware,這樣并不會影響整個流程
class CollectionMiddleware(MiddlewareMixin):
def process_request(self, request):
log.info('進入CollectionMiddleware,收集請求數據')
request.META['REQUEST_BODY'] = json.loads(str(request.body, encoding='utf-8').replace(' ', '').replace('\n',
'').replace(
'\t',
''))
if 'HTTP_X_FORWARDED_FOR' in request.META:
remote_address = request.META['HTTP_X_FORWARDED_FOR']
else:
remote_address = request.META['REMOTE_ADDR']
request.META['IP'] = remote_address
request.META['LOG_UUID'] = gen_uuid()
def process_response(self, request, response):
log.info('進入CollectionMiddleware,收集響應數據')
# 獲取請求的 uid,如果是未登錄的則為 None
if not isinstance(request.user, AnonymousUser):
uid = request.user.uid
else:
uid = None
request.META['USER_UID'] = uid
# 獲取響應內容
if response['content-type'] == 'application/json':
if getattr(response, 'streaming', False):
response_body = '<<<Streaming>>>'
else:
response_body = json.loads(str(response.content, encoding='utf-8'))
else:
response_body = '<<<Not JSON>>>'
request.META['RESP_BODY'] = response_body
# 獲取請求的 view 視圖名稱
try:
request.META['VIEW'] = request.resolver_match.view_name
except AttributeError:
request.META['VIEW'] = None
request.META['STATUS_CODE'] = response.status_code
# 設置 headers: X-Log-Id
response.setdefault('X-Log-Id', request.META['LOG_UUID'])
return response
中間件注冊順序
到現在,有4個中間件了
CollectionMiddleware
收集請求、響應數據,分別在 process_request() 和process_response()中執行
LoggerMiddleware
寫日志,在 process_response()中執行
SentryMiddleware
上報請求數據,在 process_response()中執行
前面說過,process_response()是從下到上順序執行,那作為收集響應數據的CollectionMiddleware,應該排在最后一個
LoggerMiddleware和SentryMiddleware都是在process_response()里面進行記錄和上報的,所以應該在CollectionMiddleware收集響應數據后再執行,那么應該在CollectionMiddleware上面。
那么初步的順序從上往下就是
LoggerMiddleware
SentryMiddleware
CollectionMiddleware
其中 LoggerMiddleware 和 SentryMiddleware 不分先后
考慮到 django 自帶有一些中間件,會處理一些數據,而我們的LoggerMiddleware和SentryMiddleware肯定是需要記錄最終的數據,那么將LoggerMiddleware和SentryMiddleware放在所有中間件最上面;
同理,CollectionMiddleware則放在最下面;
MIDDLEWARE = [
'buggg.core.middlewares.LoggerMiddleware',
'buggg.core.middlewares.SentryMiddleware',
'corsheaders.middleware.CorsMiddleware',
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
# 'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'buggg.core.middlewares.CollectionMiddleware'
]
演示
我們來看下實際調用的演示
Postman 發送請求
<img src="https://gitee.com/slowchen/img_bed/raw/master/uPic/2021/01/image-20210131123358918.png" alt="image-20210131123358918" style="zoom:50%;" />
控制臺日志輸出
<img src="https://gitee.com/slowchen/img_bed/raw/master/uPic/2021/01/image-20210131123434909.png" alt="image-20210131123434909" style="zoom:50%;" />
這個json 串如下:
::: details 展開查看 json
{
"request":{
"uid":"unu4cx915z4wkbnw",
"ip":"127.0.0.1",
"method":"POST",
"path":"/api/user/role",
"view":"apps.user.views.RoleView",
"body":{
"role_name":"super1",
"role_code":"SUPER1",
"role_desc":"superadminrole"
},
"headers":{
"authorization":"Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpYXQiOjE2MTE5NzUwMTUsImV4cCI6NDc2Nzc1NjYxNSwic3ViIjoiUkFJTkJPVyIsInVpZCI6InVudTRjeDkxNXo0d2tibncifQ.uFM1LOkb01SxQXXRF4q03jAi753Q-BntcEpJ-tTCX5Y",
"user_agent":"PostmanRuntime/7.26.8",
"accept":"*/*",
"postman_token":"d1f825ba-cadb-4b0e-a31f-d49bb2b5e033",
"host":"0.0.0.0:8000",
"accept_encoding":"gzip, deflate, br",
"connection":"keep-alive"
}
},
"response":{
"status":200,
"body":{
"code":20200,
"msg":"創建成功",
"data":{
"id":3,
"role_name":"super1",
"role_code":"SUPER1",
"role_desc":"super admin role"
},
"timestamp":1612067614
},
"headers":{
"Content-Type":"application/json",
"Vary":"Accept, Origin",
"Allow":"GET, POST, HEAD, OPTIONS",
"X-Log-Id":"7a6a233e637d11eb9f36acde48001122",
"X-Frame-Options":"DENY",
"Content-Length":"147",
"X-Content-Type-Options":"nosniff",
"Referrer-Policy":"same-origin"
}
},
"log_uuid":"7a6a233e637d11eb9f36acde48001122"
}
:::
接口響應 headers 數據,有我們自定義的 x-log-id
<img src="https://gitee.com/slowchen/img_bed/raw/master/uPic/2021/01/image-20210131123632415.png" alt="image-20210131123632415" style="zoom:50%;" />
sentry 頁面顯示
<img src="https://gitee.com/slowchen/img_bed/raw/master/uPic/2021/01/image-20210131123952528.png" alt="image-20210131123952528" style="zoom:50%;" />
總結
至此,通過中間件記錄日志,上報 sentry 就完成了,這種方式的好處是,各個中間件各司其職,而且可以方便以后增加中間件,不會有影響,也不會有冗余代碼,個人覺得是我能想到的稍微好點的處理方式。
當然,以上只是我個人的看法,如果你有更好的解決方案,歡迎討論。
最終代碼
::: details 中間件代碼
import json
import sentry_sdk
from django.contrib.auth.models import AnonymousUser
from django.utils.deprecation import MiddlewareMixin
from buggg.utils.log import log
from buggg.utils.utils import gen_uuid
def _get_request_headers(request):
headers = {}
for k, v in request.META.items():
if k.startswith('HTTP_'):
headers[k[5:].lower()] = v
return headers
def _get_response_headers(response):
headers = {}
headers_tuple = response.items()
for i in headers_tuple:
headers[i[0]] = i[1]
return headers
NOT_SUPPORT_PATH = '/admin' # 排除 admin 站點,admin 站點不會進入CollectionMiddleware的process_response方法,會導致報錯
class CollectionMiddleware(MiddlewareMixin):
def process_request(self, request):
# 增加判斷,如果請求的 path 是以/admin/開頭的,則直接放過,不做任何處理
if request.path.startswith(NOT_SUPPORT_PATH):
pass
else:
log.info('進入CollectionMiddleware,收集請求數據')
request.META['REQUEST_BODY'] = json.loads(str(request.body, encoding='utf-8').replace(' ', '').replace('\n',
'').replace(
'\t',
''))
if 'HTTP_X_FORWARDED_FOR' in request.META:
remote_address = request.META['HTTP_X_FORWARDED_FOR']
else:
remote_address = request.META['REMOTE_ADDR']
request.META['IP'] = remote_address
request.META['LOG_UUID'] = gen_uuid()
def process_response(self, request, response):
# 增加判斷,如果請求的 path 是以/admin/開頭的,則直接放過,不做任何處理
if request.path.startswith(NOT_SUPPORT_PATH):
pass
else:
log.info('進入CollectionMiddleware,收集響應數據')
# 獲取請求的 uid,如果是未登錄的則為 None
if not isinstance(request.user, AnonymousUser):
uid = request.user.uid
else:
uid = None
request.META['USER_UID'] = uid
# 獲取響應內容
if response['content-type'] == 'application/json':
if getattr(response, 'streaming', False):
response_body = '<<<Streaming>>>'
else:
response_body = json.loads(str(response.content, encoding='utf-8'))
else:
response_body = '<<<Not JSON>>>'
request.META['RESP_BODY'] = response_body
# 獲取請求的 view 視圖名稱
try:
request.META['VIEW'] = request.resolver_match.view_name
except AttributeError:
request.META['VIEW'] = None
request.META['STATUS_CODE'] = response.status_code
# 設置 headers: X-Log-Id
response.setdefault('X-Log-Id', request.META['LOG_UUID'])
return response
class LoggerMiddleware(MiddlewareMixin):
"""
中間件,記錄日志
"""
def process_request(self, request):
pass
def process_response(self, request, response):
# 增加判斷,如果請求的 path 是以/admin/開頭的,則直接放過,不做任何處理
if request.path.startswith(NOT_SUPPORT_PATH):
pass
else:
log.info('進入LoggerMiddleware,寫日志')
log_data = {
'request': {
'uid': request.META['USER_UID'],
"ip": request.META['IP'],
"method": request.method,
'path': request.get_full_path(),
'view': request.META['VIEW'],
'body': request.META['REQUEST_BODY'],
'headers': _get_request_headers(request),
},
'response': {
'status': request.META['STATUS_CODE'],
'body': request.META['RESP_BODY'],
'headers': _get_response_headers(response),
},
'log_uuid': request.META['LOG_UUID']
}
log.info(json.dumps(log_data))
return response
class SentryMiddleware(MiddlewareMixin):
"""
上報 sentry
"""
def process_request(self, request):
pass
def process_response(self, request, response):
# 增加判斷,如果請求的 path 是以/admin/開頭的,則直接放過,不做任何處理
if request.path.startswith(NOT_SUPPORT_PATH):
pass
else:
log.info('進入SentryMiddleware,上報請求至Sentry')
sentry_sdk.add_breadcrumb(
category='path',
message=request.path,
level='info',
)
sentry_sdk.add_breadcrumb(
category='body',
message=request.META["REQUEST_BODY"],
level='info',
)
sentry_sdk.add_breadcrumb(
category='request_headers',
message=_get_request_headers(request),
level='info',
)
sentry_sdk.add_breadcrumb(
category='response_headers',
message=_get_response_headers(response),
level='info',
)
sentry_sdk.add_breadcrumb(
category='view',
message=request.META['VIEW'],
level='info',
)
sentry_sdk.set_user({"id": request.META['USER_UID']})
sentry_sdk.set_tag("log-id", request.META["LOG_UUID"])
sentry_sdk.capture_message(request.META["LOG_UUID"])
return response
:::
個人博客
我的個人博客在這里哦~