1. 前言
在傳統單體服務
或少數微服務架構
中,debug相對簡單,微服務之間的依賴關系也比較容易梳理。
但是,在較大的微服務架構
中,如何更好地解決上述問題,需要引入微服務調用鏈,來記錄和排查問題,比如Zipkin。
主要解決的問題:
- 微服務系統瓶頸分析,通過分析API調用鏈,了解哪個是核心service(需提高健壯性),那個service的耗時較長(需拆分或代碼優化),等
- Debug,troubleshoot latency problems,or find Exception in a data flow.
-
微服務依賴關系,zipkin提供服務依賴關系分析功能
微服務架構
2. zipkin架構
2.1. components
collector
Once the trace data arrives at the Zipkin collector daemon, it is validated, stored, and indexed for lookups by the Zipkin collector.
storage
Zipkin was initially built to store data on Cassandra since Cassandra is scalable, has a flexible schema, and is heavily used within Twitter. However, we made this component pluggable. In addition to Cassandra, we natively support ElasticSearch and MySQL. Other back-ends might be offered as third party extensions.
search
Once the data is stored and indexed, we need a way to extract it. The query daemon provides a simple JSON API for finding and retrieving traces. The primary consumer of this API is the Web UI.
https://zipkin.io/zipkin-api/
web UI
We created a GUI that presents a nice interface for viewing traces. The web UI provides a method for viewing traces based on service, time, and annotations.
Note: there is no built-in authentication in the UI!
2.2. Terms
transport
Spans sent by the instrumented library must be transported from the services being traced to Zipkin collectors. There are three primary transports: HTTP, Kafka and Scribe.
trace
trace表示一個api調用鏈,即經過哪些service,每個service消耗的時間,等
span
span表示每次調用,即參與api調用鏈的一個service。主要包含三個基本元素,span_id, parenet_id, trace_id,其他信息還包含,name、duration、tags、localEndpoint等
{
"traceId": "5982fe77008310cc80f1da5e10147517",
"name": "query",
"id": "be2d01e33cc78d97",
"parentId": "ebf33e1a81dc6f71",
"timestamp": 1458702548786000,
"duration": 13000,
"localEndpoint": {
"serviceName": "zipkin-query",
"ipv4": "192.168.1.2",
"port": 9411
},
"remoteEndpoint": {
"serviceName": "spanstore-jdbc",
"ipv4": "127.0.0.1",
"port": 3306
},
"annotations": [
{
"timestamp": 1458702548786000,
"value": "cs"
},
{
"timestamp": 1458702548799000,
"value": "cr"
}
],
"tags": {
"jdbc.query": "select distinct `zipkin_spans`.`trace_id` from `zipkin_spans` join `zipkin_annotations` on (`zipkin_spans`.`trace_id` = `zipkin_annotations`.`trace_id` and `zipkin_spans`.`id` = `zipkin_annotations`.`span_id`) where (`zipkin_annotations`.`endpoint_service_name` = ? and `zipkin_spans`.`start_ts` between ? and ?) order by `zipkin_spans`.`start_ts` desc limit ?",
"sa": "true"
}
}
2.3. 工作原理
- 注意:需要異步發送span,否則,會影響業務API
- 支持的語言或框架
https://zipkin.io/pages/tracers_instrumentation
3. 啟動zipkin
- zipkin提供了多種啟動方式,java、docker、docker-compose等
docker run -d -p 9411:9411 openzipkin/zipkin
-
訪問zipkin ui, http://localhost:9411
zipkin ui 更多啟動方式
https://hub.docker.com/r/openzipkin/zipkin/
https://github.com/openzipkin/zipkin/tree/master/docker
4. flask+zipkin
- start multiple service
$ SERVICE_NAME=app3 SERVICE_PORT=6002 \
python app.py
$ SERVICE_NAME=app2 SERVICE_PORT=6001 NEXT_SERVICE_API=http://localhost:6002 \
python app.py
$ SERVICE_NAME=app1 SERVICE_PORT=6000 NEXT_SERVICE_API=http://localhost:6001 \
python app.py
- call api portal
$ http :6000/hello
HTTP/1.0 200 OK
Content-Length: 5
Content-Type: text/html; charset=utf-8
Date: Sun, 14 Jun 2020 05:14:33 GMT
Server: Werkzeug/1.0.1 Python/3.6.9
hello
- check zipkin ui
http://localhost:9411
image.png
image.png
5. code
- app.py
from flask import Flask, request, abort, g
from py_zipkin.zipkin import create_http_headers_for_new_span
import requests
from time import sleep
from random import randint
import config as global_config
from logging_helper import logger
from zipkin import flask_start_zipkin, flask_stop_zipkin, set_tags
def create_app(config=None):
app = Flask(__name__)
config_app(app, config)
init_controller(app)
@app.before_request
def before():
flask_start_zipkin()
@app.teardown_request
def teardown(e):
flask_stop_zipkin()
return app
def init_controller(app):
@app.route('/<string:name>', methods=['GET'])
def do_stuff(name):
logger.info(f'{global_config.SERVICE_NAME} - {name}')
sleep(randint(100, 1000)/1000)
if global_config.NEXT_SERVICE_API:
headers = create_http_headers_for_new_span()
requests.get(f'{global_config.NEXT_SERVICE_API}/{name}', headers=headers)
set_tags(dict(api_name=name))
return name, 200
def config_app(app, config):
app.config.from_object(global_config)
if config is not None and isinstance(config, dict):
app.config.update(config)
if __name__ == '__main__':
app = create_app()
app.run(host="0.0.0.0", port=global_config.SERVICE_PORT, debug=True)
- config.py
import os
env = os.environ.get
SERVICE_NAME = env('SERVICE_NAME', 'try_zipkin')
SERVICE_PORT = env('SERVICE_PORT', 6000)
NEXT_SERVICE_API = env('NEXT_SERVICE_API')
# ===== Zipkin ========
ZIPKIN_HOST = env('ZIPKIN_HOST', '127.0.0.1')
ZIPKIN_PORT = env('ZIPKIN_PORT', 9411)
ZIPKIN_SAMPLE_RATE = env('ZIPKIN_SAMPLE_RATE', 100.0)
ZIPKIN_DISABLE = env("ZIPKIN_DISABLE", False)
ZIPKIN_SERVICE_NAME = SERVICE_NAME
- zipkin.py
import random
import functools
from flask import request, g
import requests
from py_zipkin.zipkin import zipkin_span, ZipkinAttrs
from py_zipkin.transport import BaseTransportHandler
import config as config
from logging_helper import logger
def gen_hex_str(length=16):
charset = (
'abcdef'
'0123456789'
)
return ''.join([random.choice(charset) for _ in range(length)])
class HttpTransport(BaseTransportHandler):
def get_max_payload_bytes(self):
return None
def send(self, encoded_span):
# The collector expects a thrift-encoded list of spans.
try:
requests.post(
f'http://{config.ZIPKIN_HOST}:{config.ZIPKIN_PORT}/api/v1/spans',
data=encoded_span,
headers={'Content-Type': 'application/x-thrift'},
timeout=(1, 5),
)
except Exception as e:
logger.exception("Failed to send to zipkin: %s", str(e))
class KafkaTransport(BaseTransportHandler):
def get_max_payload_bytes(self):
# By default Kafka rejects messages bigger than 1000012 bytes.
return 1000012
def send(self, message):
from kafka import SimpleProducer, KafkaClient
kafka_client = KafkaClient('{}:{}'.format('localhost', 9092))
producer = SimpleProducer(kafka_client)
producer.send_messages('kafka_topic_name', message)
def with_zipkin_span(span_name, **kwargs):
"""https://github.com/Yelp/py_zipkin/issues/96
"""
def decorate(f):
@functools.wraps(f)
def inner(*args, **kw):
zipkin_kwargs = dict(
span_name=span_name,
service_name=config.ZIPKIN_SERVICE_NAME
)
zipkin_kwargs.update(**kwargs)
with zipkin_span(**zipkin_kwargs):
return f(*args, **kw)
return inner
return decorate
def set_tags(extra_annotations):
if getattr(g, '_zipkin_span', None):
logger.info("Set tags")
try:
span = g._zipkin_span
if span:
logger.info("update: %s", extra_annotations)
span.update_binary_annotations(extra_annotations)
except Exception as e:
logger.log_critical_error("Failed to set zipkin: %s", str(e))
def get_zipkin_attrs(headers):
trace_id = headers.get('X-B3-TraceID') or gen_hex_str()
span_id = headers.get('X-B3-SpanID') or gen_hex_str()
parent_span_id = headers.get('X-B3-ParentSpanID')
flags = headers.get('X-B3-Flags')
is_sampled = headers.get('X-B3-Sampled') == '1'
return ZipkinAttrs(
trace_id=trace_id,
span_id=span_id,
parent_span_id=parent_span_id,
flags=flags,
is_sampled=is_sampled,
)
def flask_start_zipkin():
"""
Put it to before_request event.
:return:
"""
if config.ZIPKIN_DISABLE:
return
zipkin_attrs = get_zipkin_attrs(request.headers)
if request.headers.get('X-B3-TraceID') or request.headers.get('X-B3-SpanID'):
logger.info(f'Zipkin Debug: received header: {request.headers}')
span = zipkin_span(
service_name=config.ZIPKIN_SERVICE_NAME,
span_name=f'{request.endpoint}.{request.method}',
transport_handler=HttpTransport(),
host=config.ZIPKIN_HOST,
port=config.ZIPKIN_PORT,
sample_rate=config.ZIPKIN_SAMPLE_RATE,
zipkin_attrs=zipkin_attrs,
)
g._zipkin_span = span
g._zipkin_span.start()
def flask_stop_zipkin():
"""
Put it to tear_request event.
:return:
"""
if config.ZIPKIN_DISABLE:
return
if getattr(g, '_zipkin_span', None):
print(f'stop zipkin span {g._zipkin_span}')
g._zipkin_span.stop()
- logging_helper.py
import logging
import types
import sys
import config as config
def init_logger_handler():
logger_ = logging.getLogger(config.SERVICE_NAME) # type: logging.Logger
logger_.setLevel(logging.INFO)
handler = logging.StreamHandler()
handler.setLevel(logging.INFO)
formatter = logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
handler.setFormatter(formatter)
logger_.addHandler(handler)
return logger_
def log_response_error(self, resp):
if sys.exc_info() != (None, None, None):
self.exception('request failed')
if resp is None:
self.error('request hasn\'t been sent out')
else:
try:
self.error(resp.json())
except Exception:
try:
self.error(resp.text)
except Exception:
self.error(f'unknown resp content (status code {resp.status_code})')
logger = init_logger_handler()
logger.log_response_error = types.MethodType(log_response_error, logger)
- requirements.txt
-i https://mirrors.aliyun.com/pypi/simple
flask
requests
py_zipkin
6. references
- 一文搞懂基于zipkin的分布式追蹤系統原理與實現
https://juejin.im/post/5c3d4df0f265da61307517ad - zipkin
https://zipkin.io/