1. 前言

在傳統單體服務或少數微服務架構中，debug相對簡單，微服務之間的依賴關系也比較容易梳理。
但是，在較大的微服務架構中，如何更好地解決上述問題，需要引入微服務調用鏈，來記錄和排查問題，比如Zipkin。
主要解決的問題：

• 微服務系統瓶頸分析，通過分析API調用鏈，了解哪個是核心service（需提高健壯性），那個service的耗時較長（需拆分或代碼優化），等
• Debug，troubleshoot latency problems，or find Exception in a data flow.

• 微服務依賴關系，zipkin提供服務依賴關系分析功能

  
  
  微服務架構

2. zipkin架構

zipkin架構

2.1. components

collector

Once the trace data arrives at the Zipkin collector daemon, it is validated, stored, and indexed for lookups by the Zipkin collector.

storage

Zipkin was initially built to store data on Cassandra since Cassandra is scalable, has a flexible schema, and is heavily used within Twitter. However, we made this component pluggable. In addition to Cassandra, we natively support ElasticSearch and MySQL. Other back-ends might be offered as third party extensions.

search

Once the data is stored and indexed, we need a way to extract it. The query daemon provides a simple JSON API for finding and retrieving traces. The primary consumer of this API is the Web UI.
https://zipkin.io/zipkin-api/

web UI

We created a GUI that presents a nice interface for viewing traces. The web UI provides a method for viewing traces based on service, time, and annotations.
Note: there is no built-in authentication in the UI!

2.2. Terms

transport

Spans sent by the instrumented library must be transported from the services being traced to Zipkin collectors. There are three primary transports: HTTP, Kafka and Scribe.

trace

trace表示一個api調用鏈，即經過哪些service，每個service消耗的時間，等

span

span表示每次調用，即參與api調用鏈的一個service。主要包含三個基本元素，span_id, parenet_id, trace_id，其他信息還包含，name、duration、tags、localEndpoint等

{
      "traceId": "5982fe77008310cc80f1da5e10147517",
      "name": "query",
      "id": "be2d01e33cc78d97",
      "parentId": "ebf33e1a81dc6f71",
      "timestamp": 1458702548786000,
      "duration": 13000,
      "localEndpoint": {
        "serviceName": "zipkin-query",
        "ipv4": "192.168.1.2",
        "port": 9411
      },
      "remoteEndpoint": {
        "serviceName": "spanstore-jdbc",
        "ipv4": "127.0.0.1",
        "port": 3306
      },
      "annotations": [
        {
          "timestamp": 1458702548786000,
          "value": "cs"
        },
        {
          "timestamp": 1458702548799000,
          "value": "cr"
        }
      ],
      "tags": {
        "jdbc.query": "select distinct `zipkin_spans`.`trace_id` from `zipkin_spans` join `zipkin_annotations` on (`zipkin_spans`.`trace_id` = `zipkin_annotations`.`trace_id` and `zipkin_spans`.`id` = `zipkin_annotations`.`span_id`) where (`zipkin_annotations`.`endpoint_service_name` = ? and `zipkin_spans`.`start_ts` between ? and ?) order by `zipkin_spans`.`start_ts` desc limit ?",
        "sa": "true"
      }
    }

2.3. 工作原理

如何使用zipkin

• 注意：需要異步發送span，否則，會影響業務API
• 支持的語言或框架
  https://zipkin.io/pages/tracers_instrumentation

3. 啟動zipkin

• zipkin提供了多種啟動方式，java、docker、docker-compose等

docker run -d -p 9411:9411 openzipkin/zipkin

• 訪問zipkin ui, http://localhost:9411
  

  zipkin ui

• 更多啟動方式
  https://hub.docker.com/r/openzipkin/zipkin/
  https://github.com/openzipkin/zipkin/tree/master/docker

4. flask+zipkin

• start multiple service

$ SERVICE_NAME=app3 SERVICE_PORT=6002 \
python app.py
$ SERVICE_NAME=app2 SERVICE_PORT=6001 NEXT_SERVICE_API=http://localhost:6002 \
python app.py
$ SERVICE_NAME=app1 SERVICE_PORT=6000 NEXT_SERVICE_API=http://localhost:6001 \
python app.py

• call api portal

$ http :6000/hello
HTTP/1.0 200 OK
Content-Length: 5
Content-Type: text/html; charset=utf-8
Date: Sun, 14 Jun 2020 05:14:33 GMT
Server: Werkzeug/1.0.1 Python/3.6.9

hello

• check zipkin ui
  http://localhost:9411
  
  image.png
  
  
  image.png

5. code

• app.py

from flask import Flask, request, abort, g
from py_zipkin.zipkin import create_http_headers_for_new_span
import requests
from time import sleep
from random import randint

import config as global_config
from logging_helper import logger
from zipkin import flask_start_zipkin, flask_stop_zipkin, set_tags


def create_app(config=None):
    app = Flask(__name__)

    config_app(app, config)
    init_controller(app)

    @app.before_request
    def before():
        flask_start_zipkin()

    @app.teardown_request
    def teardown(e):
        flask_stop_zipkin()

    return app


def init_controller(app):
    @app.route('/<string:name>', methods=['GET'])
    def do_stuff(name):
        logger.info(f'{global_config.SERVICE_NAME} - {name}')
        sleep(randint(100, 1000)/1000)
        if global_config.NEXT_SERVICE_API:
            headers = create_http_headers_for_new_span()
            requests.get(f'{global_config.NEXT_SERVICE_API}/{name}', headers=headers)
            set_tags(dict(api_name=name))
        return name, 200


def config_app(app, config):
    app.config.from_object(global_config)

    if config is not None and isinstance(config, dict):
        app.config.update(config)


if __name__ == '__main__':
    app = create_app()
    app.run(host="0.0.0.0", port=global_config.SERVICE_PORT, debug=True)

• config.py

import os

env = os.environ.get
SERVICE_NAME = env('SERVICE_NAME', 'try_zipkin')
SERVICE_PORT = env('SERVICE_PORT', 6000)
NEXT_SERVICE_API = env('NEXT_SERVICE_API')

# ===== Zipkin ========
ZIPKIN_HOST = env('ZIPKIN_HOST', '127.0.0.1')
ZIPKIN_PORT = env('ZIPKIN_PORT', 9411)
ZIPKIN_SAMPLE_RATE = env('ZIPKIN_SAMPLE_RATE', 100.0)
ZIPKIN_DISABLE = env("ZIPKIN_DISABLE", False)
ZIPKIN_SERVICE_NAME = SERVICE_NAME

• zipkin.py

import random
import functools
from flask import request, g
import requests
from py_zipkin.zipkin import zipkin_span, ZipkinAttrs
from py_zipkin.transport import BaseTransportHandler

import config as config
from logging_helper import logger


def gen_hex_str(length=16):
    charset = (
        'abcdef'
        '0123456789'
    )
    return ''.join([random.choice(charset) for _ in range(length)])


class HttpTransport(BaseTransportHandler):
    def get_max_payload_bytes(self):
        return None

    def send(self, encoded_span):
        # The collector expects a thrift-encoded list of spans.
        try:
            requests.post(
                f'http://{config.ZIPKIN_HOST}:{config.ZIPKIN_PORT}/api/v1/spans',
                data=encoded_span,
                headers={'Content-Type': 'application/x-thrift'},
                timeout=(1, 5),
            )
        except Exception as e:
            logger.exception("Failed to send to zipkin: %s", str(e))


class KafkaTransport(BaseTransportHandler):

    def get_max_payload_bytes(self):
        # By default Kafka rejects messages bigger than 1000012 bytes.
        return 1000012

    def send(self, message):
        from kafka import SimpleProducer, KafkaClient
        kafka_client = KafkaClient('{}:{}'.format('localhost', 9092))
        producer = SimpleProducer(kafka_client)
        producer.send_messages('kafka_topic_name', message)


def with_zipkin_span(span_name, **kwargs):
    """https://github.com/Yelp/py_zipkin/issues/96
    """
    def decorate(f):
        @functools.wraps(f)
        def inner(*args, **kw):
            zipkin_kwargs = dict(
                span_name=span_name,
                service_name=config.ZIPKIN_SERVICE_NAME
            )
            zipkin_kwargs.update(**kwargs)
            with zipkin_span(**zipkin_kwargs):
                return f(*args, **kw)
        return inner
    return decorate


def set_tags(extra_annotations):
    if getattr(g, '_zipkin_span', None):
        logger.info("Set tags")
        try:
            span = g._zipkin_span
            if span:
                logger.info("update: %s", extra_annotations)
                span.update_binary_annotations(extra_annotations)
        except Exception as e:
            logger.log_critical_error("Failed to set zipkin: %s", str(e))


def get_zipkin_attrs(headers):
    trace_id = headers.get('X-B3-TraceID') or gen_hex_str()
    span_id = headers.get('X-B3-SpanID') or gen_hex_str()
    parent_span_id = headers.get('X-B3-ParentSpanID')
    flags = headers.get('X-B3-Flags')
    is_sampled = headers.get('X-B3-Sampled') == '1'

    return ZipkinAttrs(
        trace_id=trace_id,
        span_id=span_id,
        parent_span_id=parent_span_id,
        flags=flags,
        is_sampled=is_sampled,
    )


def flask_start_zipkin():
    """
    Put it to before_request event.
    :return:
    """
    if config.ZIPKIN_DISABLE:
        return
    zipkin_attrs = get_zipkin_attrs(request.headers)
    if request.headers.get('X-B3-TraceID') or request.headers.get('X-B3-SpanID'):
        logger.info(f'Zipkin Debug: received header: {request.headers}')
    span = zipkin_span(
        service_name=config.ZIPKIN_SERVICE_NAME,
        span_name=f'{request.endpoint}.{request.method}',
        transport_handler=HttpTransport(),
        host=config.ZIPKIN_HOST,
        port=config.ZIPKIN_PORT,
        sample_rate=config.ZIPKIN_SAMPLE_RATE,
        zipkin_attrs=zipkin_attrs,
    )
    g._zipkin_span = span
    g._zipkin_span.start()


def flask_stop_zipkin():
    """
    Put it to tear_request event.
    :return:
    """
    if config.ZIPKIN_DISABLE:
        return

    if getattr(g, '_zipkin_span', None):
        print(f'stop zipkin span {g._zipkin_span}')
        g._zipkin_span.stop()

• logging_helper.py

import logging
import types
import sys

import config as config


def init_logger_handler():
    logger_ = logging.getLogger(config.SERVICE_NAME)  # type: logging.Logger

    logger_.setLevel(logging.INFO)

    handler = logging.StreamHandler()
    handler.setLevel(logging.INFO)
    formatter = logging.Formatter(
        '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
    )
    handler.setFormatter(formatter)
    logger_.addHandler(handler)
    return logger_


def log_response_error(self, resp):
    if sys.exc_info() != (None, None, None):
        self.exception('request failed')

    if resp is None:
        self.error('request hasn\'t been sent out')
    else:
        try:
            self.error(resp.json())
        except Exception:
            try:
                self.error(resp.text)
            except Exception:
                self.error(f'unknown resp content (status code {resp.status_code})')


logger = init_logger_handler()
logger.log_response_error = types.MethodType(log_response_error, logger)

• requirements.txt

-i https://mirrors.aliyun.com/pypi/simple
flask
requests
py_zipkin

6. references

• 一文搞懂基于zipkin的分布式追蹤系統原理與實現
  https://juejin.im/post/5c3d4df0f265da61307517ad
• zipkin
  https://zipkin.io/