Kolla-ansible 源碼分析

簡介

Kolla-ansible項目提供一個完整的Ansible Playbook，來部署Docker的鏡像，再完成openstack組件的自動化部署。并提供all-in-one和multihost的環(huán)境。
源碼地址：https://github.com/openstack/kolla-ansible.git

源碼目錄概要

一級目錄

Paste_Image.png

• Ansible: ansible的整個playbook代碼,包括部署docker容器和openstack組件。源碼主要集中在這個目錄下。
• Contrib:包括用heat和magnum的部署環(huán)境和vagrant的部署環(huán)境。
• Deploy-guide: 部署指南，主要包括all-in-one和mulihosts兩種部署方式的指南。
• Doc:文檔。
• Etc: 一些配置文件，安裝完了引用到了/etc目錄下，all-in-one只要修改很少量的配置。
• Kolla-ansible: 記錄版本信息，cmd子目錄下有生成密碼和合并密碼的兩個腳本。pbr打包的時候被封裝成為可執(zhí)行命令。
• Releasenodes: 發(fā)布特性說明。
• Specs: 包含有Kolla社區(qū)關(guān)鍵參數(shù)代碼庫的變化。
• Tests: 包括一些功能測試工具，這里還包括兩個自定義的ansible plugin（merge_config）和module（kolla_docker）的測試。
• Tools: 一些和kolla交換的腳本工具，大部分是可手動調(diào)用，主要完成一些安裝前后的一些操作。有些會被ansible目錄下的task調(diào)用到。

二級目錄

• Ansible/action_plugins: 自定義ansible插件，兩個腳本，用于合并yml和conifg的配置文件。
• Ansible/group_vars: ansible腳本的全局變量定義。
• Ansible/inventory: 包含all-in-one和mulitnode的樣板hosts清單。
• Ansible/library: 包括一些自定義的ansible模塊，bslurp.py和kolla_docker.py用到比較多。
• Ansible/role: 所有的openstack的組件，幾乎包含了說有開源項目，當前版本有60個組件。
• Ansible:除了文件夾之外的ansible腳本，主要用戶安裝前后的環(huán)境準備和清理，數(shù)據(jù)庫恢復(fù)等特殊系統(tǒng)級的操作。

關(guān)鍵代碼解讀

Setup.cfg安裝配置入口文件， 見中文注釋

[metadata]
name = kolla-ansible  // 項目名稱
summary = Ansible Deployment of Kolla containers
description-file = README.rst
author = OpenStack
author-email = openstack-dev@lists.openstack.org
home-page = http://docs.openstack.org/developer/kolla-ansible/
license = Apache License, Version 2.0
classifier =
    Environment :: OpenStack
    Intended Audience :: Information Technology
    Intended Audience :: System Administrators
    License :: OSI Approved :: Apache Software License
    Operating System :: POSIX :: Linux
    Programming Language :: Python
    Programming Language :: Python :: 2
    Programming Language :: Python :: 2.7
    Programming Language :: Python :: 3
    Programming Language :: Python :: 3.5

 [files]
packages = kolla_ansible   //包名
data_files =        //pbr方式打包對應(yīng)的文件映射
    share/kolla-ansible/ansible = ansible/*
    share/kolla-ansible/tools = tools/validate-docker-execute.sh
    share/kolla-ansible/tools = tools/cleanup-containers
    share/kolla-ansible/tools = tools/cleanup-host
    share/kolla-ansible/tools = tools/cleanup-images
    share/kolla-ansible/tools = tools/stop-containers
share/kolla-ansible/doc = doc/*
share/kolla-ansible/etc_examples = etc/*
share/kolla-ansible = tools/init-runonce
share/kolla-ansible = tools/init-vpn
share/kolla-ansible = tools/openrc-example
share/kolla-ansible = setup.cfg

scripts =        //可執(zhí)行腳本
    tools/kolla-ansible

[entry_points]
console_scripts =   //控制臺可執(zhí)行腳本，執(zhí)行兩個Python文件的main函數(shù)
kolla-genpwd = kolla_ansible.cmd.genpwd:main
kolla-mergepwd = kolla_ansible.cmd.mergepwd:main

[global]
setup-hooks =
pbr.hooks.setup_hook

[pbr]  //打包方式

[build_sphinx]
all_files = 1
build-dir = doc/build
source-dir = doc

[build_releasenotes]
all_files = 1
build-dir = releasenotes/build
source-dir = releasenotes/source

Setup.py

安裝執(zhí)行腳本，通過pbr打包，執(zhí)行過程會讀取setup.cfg配置，還會安裝同父目錄下requirements.txt中的依賴。更多參考https://julien.danjou.info/blog/2017/packaging-python-with-pbr

import setuptools

# In python < 2.7.4, a lazy loading of package `pbr` will break
# setuptools if some other modules registered functions in `atexit`.
# solution from: http://bugs.python.org/issue15881#msg170215
try:
    import multiprocessing  # noqa
except ImportError:
    pass

setuptools.setup(
    setup_requires=['pbr>=2.0.0'],
    pbr=True)

tools\kolla-ansible

該腳本是封裝了ansible-playbook，對kolla進行了ansible的定制。主要根據(jù)action的類型，傳遞不同的配置文件。
中間基礎(chǔ)變量定義：

find_base_dir
INVENTORY="${BASEDIR}/ansible/inventory/all-in-one"
PLAYBOOK="${BASEDIR}/ansible/site.yml"
VERBOSITY=
EXTRA_OPTS=${EXTRA_OPTS}
CONFIG_DIR="/etc/kolla"
PASSWORDS_FILE="${CONFIG_DIR}/passwords.yml"
DANGER_CONFIRM=
INCLUDE_IMAGES=

Find_base_dir是一個腳本開始時候的一個函數(shù)（不展開解釋），用于找到kolla-ansible腳本所在的路徑。

腳本傳參解釋：
while [ "$#" -gt 0 ]; do
case "$1" in

    (--inventory|-i)
            INVENTORY="$2"
            shift 2
            ;;

    (--playbook|-p)
            PLAYBOOK="$2"
            shift 2
            ;;

    (--tags|-t)
            EXTRA_OPTS="$EXTRA_OPTS --tags $2"
            shift 2
            ;;

    (--verbose|-v)
            VERBOSITY="$VERBOSITY --verbose"
            shift 1
            ;;

    (--configdir)
            CONFIG_DIR="$2"
            shift 2
            ;;

    (--yes-i-really-really-mean-it)
            DANGER_CONFIRM="$1"
            shift 1
            ;;

    (--include-images)
            INCLUDE_IMAGES="$1"
            shift 1
            ;;

    (--key|-k)
            VAULT_PASS_FILE="$2"
            EXTRA_OPTS="$EXTRA_OPTS --vault-password-file=$VAULT_PASS_FILE"
            shift 2
            ;;

    (--extra|-e)
            EXTRA_OPTS="$EXTRA_OPTS -e $2"
            shift 2
            ;;
    (--passwords)
            PASSWORDS_FILE="$2"
            shift 2
            ;;
    (--help|-h)
            usage
            shift
            exit 0
            ;;

    (--)
            shift
            break
            ;;

    (*)
            echo "error"
            exit 3
            ;;
esac
done

case "$1" in

(prechecks)
        ACTION="Pre-deployment checking"
        EXTRA_OPTS="$EXTRA_OPTS -e action=precheck"
        ;;
(check)
        ACTION="Post-deployment checking"
        EXTRA_OPTS="$EXTRA_OPTS -e action=check"
        ;;
(mariadb_recovery)
        ACTION="Attempting to restart mariadb cluster"
        EXTRA_OPTS="$EXTRA_OPTS -e action=deploy -e common_run=true"
        PLAYBOOK="${BASEDIR}/ansible/mariadb_recovery.yml"
        ;;
(destroy)
        ACTION="Destroy Kolla containers, volumes and host configuration"
        PLAYBOOK="${BASEDIR}/ansible/destroy.yml"
        if [[ "${INCLUDE_IMAGES}" == "--include-images" ]]; then
            EXTRA_OPTS="$EXTRA_OPTS -e destroy_include_images=yes"
        fi
        if [[ "${DANGER_CONFIRM}" != "--yes-i-really-really-mean-it" ]]; then
            cat << EOF
WARNING:
    This will PERMANENTLY DESTROY all deployed kolla containers, volumes and host configuration.
    There is no way to recover from this action. To confirm, please add the following option:
    --yes-i-really-really-mean-it
EOF
            exit 1
        fi
        ;;
(bootstrap-servers)
        ACTION="Bootstraping servers"
        PLAYBOOK="${BASEDIR}/ansible/kolla-host.yml"
        EXTRA_OPTS="$EXTRA_OPTS -e action=bootstrap-servers"
        ;;
(deploy)
        ACTION="Deploying Playbooks"
        EXTRA_OPTS="$EXTRA_OPTS -e action=deploy"
        ;;
(deploy-bifrost)
        ACTION="Deploying Bifrost"
        PLAYBOOK="${BASEDIR}/ansible/bifrost.yml"
        EXTRA_OPTS="$EXTRA_OPTS -e action=deploy"
        ;;
(deploy-servers)
        ACTION="Deploying servers with bifrost"
        PLAYBOOK="${BASEDIR}/ansible/bifrost.yml"
        EXTRA_OPTS="$EXTRA_OPTS -e action=deploy-servers"
        ;;
(post-deploy)
        ACTION="Post-Deploying Playbooks"
        PLAYBOOK="${BASEDIR}/ansible/post-deploy.yml"
        ;;
(pull)
        ACTION="Pulling Docker images"
        EXTRA_OPTS="$EXTRA_OPTS -e action=pull"
        ;;
(upgrade)
        ACTION="Upgrading OpenStack Environment"
        EXTRA_OPTS="$EXTRA_OPTS -e action=upgrade -e serial=${ANSIBLE_SERIAL}"
        ;;
(reconfigure)
        ACTION="Reconfigure OpenStack service"
        EXTRA_OPTS="$EXTRA_OPTS -e action=reconfigure -e serial=${ANSIBLE_SERIAL}"
        ;;
(stop)
        ACTION="Stop Kolla containers"
        PLAYBOOK="${BASEDIR}/ansible/stop.yml"
        ;;
(certificates)
        ACTION="Generate TLS Certificates"
        PLAYBOOK="${BASEDIR}/ansible/certificates.yml"
        ;;
(genconfig)
        ACTION="Generate configuration files for enabled OpenStack services"
        EXTRA_OPTS="$EXTRA_OPTS -e action=config"
        ;;
(*)     usage
        exit 0
        ;;
esac                                                                

這段是根據(jù)傳遞的參數(shù)，不同的參數(shù)針對不同的配置文件等額外屬性。這里第一個參數(shù)有好多action，如deploy，post-deploy，stop等等。

最后三行，組合命令，執(zhí)行：

CONFIG_OPTS="-e @${CONFIG_DIR}/globals.yml -e @${PASSWORDS_FILE} -e CONFIG_DIR=${CONFIG_DIR}"
CMD="ansible-playbook -i $INVENTORY $CONFIG_OPTS $EXTRA_OPTS $PLAYBOOK $VERBOSITY"
process_cmd

傳遞進來的參數(shù)組合成ansible-playbook的CMD命令后，調(diào)用process_cmd函數(shù)執(zhí)行。

• 例子1：初始命令
  kolla-ansible deploy -i /home/all-in-one
  封裝后命令
  ansible-playbook -i /home/all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e action=deploy /usr/share/kolla-ansible/ansible/site.yml
• 例子2：初始命令
  kolla-ansible post-deploy
  封裝后命令
  ansible-playbook -i /usr/share/kolla-ansible/ansible/inventory/all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla /usr/share/kolla-ansible/ansible/post-deploy.yml

ansible劇本代碼解

由于openstack的組件比較多，且大多數(shù)處于并列關(guān)系。這里不再一一展開，以neutron為例進行解讀。其他沒有涉及的有重要的點，會進行內(nèi)容穿插。

Ansible/library/

該目錄下是一些自定義的一些模塊，這些module在目標節(jié)點上運行，包括bslurp.py，kolla_container_facts.py，kolla_docker.py，kolla-toolbox.py，merge_configs.py，merge_yaml.py，前四個我會一一介紹，后兩個是空文件，代碼其實在action_plugins目錄中（那兩個空文件是映射同名的action,我們可以向module一樣使用action）。

Bslurp.py，好像做了文件分發(fā)的事情，通過copy_from_host（我沒查到調(diào)用它的地方）和copy_to_host（在ceph角色部署的時候，下發(fā)keyring到osd節(jié)點是通過這個模塊函數(shù)下發(fā)的）兩個函數(shù)實現(xiàn)，。判斷依據(jù)是模塊參數(shù)dest。見代碼中文注釋。

def copy_from_host(module):
    #此處省略不少代碼
    module.exit_json(content=base   Ａ64.b64encode(data), sha1=sha1, mode=mode,
                     source=src)

def copy_to_host(module):
    compress = module.params.get('compress')
    dest = module.params.get('dest')
    mode = int(module.params.get('mode'), 0)
    sha1 = module.params.get('sha1')
    # src是加密后的數(shù)據(jù)
    src = module.params.get('src')
    # decode已經(jīng)加密的數(shù)據(jù)
    data = base64.b64decode(src) 
#解壓數(shù)據(jù)
    raw_data = zlib.decompress(data) if compress else data

    #sha1安全算法數(shù)據(jù)校驗
    if sha1:
        if os.path.exists(dest):
            if os.access(dest, os.R_OK):
                with open(dest, 'rb') as f:
                    if hashlib.sha1(f.read()).hexdigest() == sha1:
                        module.exit_json(changed=False)
            else:
                module.exit_json(failed=True, changed=False,
                                 msg='file is not accessible: {}'.format(dest))

        if sha1 != hashlib.sha1(raw_data).hexdigest():
            module.exit_json(failed=True, changed=False,
                             msg='sha1 sum does not match data')
    
# 保存數(shù)據(jù)到dest值。這段代碼有健壯性問題，沒有考慮到磁盤寫滿的場景，保險的做法創(chuàng)建一個tmp文件，把數(shù)據(jù)拷貝到tmp文件，再把tmp文件重命名為dest值。否則容易把文件寫空。
    with os.fdopen(os.open(dest, os.O_WRONLY | os.O_CREAT, mode), 'wb') as f:
        f.write(raw_data)
    #調(diào)用module要求的exit_json接口退出。
    module.exit_json(changed=True)


def main():
# 定義dict類型的參數(shù)，ansible module的接口要求
    argument_spec = dict(
        compress=dict(default=True, type='bool'),
        dest=dict(type='str'),
        mode=dict(default='0644', type='str'),
        sha1=dict(default=None, type='str'),
        src=dict(required=True, type='str')
    )
# 創(chuàng)建ansible模塊對象
    module = AnsibleModule(argument_spec)
    # 獲取模塊dest參數(shù)值
    dest = module.params.get('dest')

    try:
        if dest: 
        # 如果dest參數(shù)存在，則推送操作，push下發(fā)到相應(yīng)的host
            copy_to_host(module)
        else:
        # 如果dest參數(shù)不存在，則進行pull操作。
            copy_from_host(module)
    except Exception:
    # 異常場景下退出，ansible自定義模塊語法規(guī)范。 
        module.exit_json(failed=True, changed=True,
                         msg=repr(traceback.format_exc()))


# import module snippets
from ansible.module_utils.basic import *  # noqa
if __name__ == '__main__':
    main()

kolla_docker.py，容器相關(guān)的操作，openstack的組件都通過容器部署，每個組件role的部署都會用到，非常重要。

...
#創(chuàng)建docker的client函數(shù)
def get_docker_client():
    try:
        return docker.Client
    except AttributeError:
        return docker.APIClient
A

class DockerWorker(object):

    def __init__(self, module):
        # 構(gòu)造函數(shù)，傳入?yún)?shù)是AnsibleModule類型的對象
        self.module = module
        # params參數(shù)續(xù)傳
        self.params = self.module.params
        self.changed = False

        # TLS not fully implemented
        # tls_config = self.generate_tls()

        # 創(chuàng)建一個docker.client對象
        options = {
            'version': self.params.get('api_version')
        }
        self.dc = get_docker_client()(**options)
# ....
# ....

# 啟動容器的函數(shù)，是AnsibleModule的其中一種action
def start_container(self):
    #檢查鏡像是否存在，不存在pull
        if not self.check_image():
            self.pull_image()
        
    #檢查容器
        container = self.check_container()
    #容器異樣，則刪除，再回調(diào)
        if container and self.check_container_differs():
            self.stop_container()
            self.remove_container()
            container = self.check_container()

        #容器不存在，創(chuàng)建，再回調(diào)
        if not container:
            self.create_container()
            container = self.check_container()
        
    #容器狀態(tài)非啟動，則啟動
        if not container['Status'].startswith('Up '):
            self.changed = True
            self.dc.start(container=self.params.get('name'))

        # We do not want to detach so we wait around for container to exit
    #如果container沒有detach斷開，那么進入wait狀態(tài),調(diào)用fail_json方法，傳遞fail的參數(shù)
    if not self.params.get('detach'):
            rc = self.dc.wait(self.params.get('name'))
            if rc != 0:
                self.module.fail_json(
                    failed=True,
                    changed=True,
                    msg="Container exited with non-zero return code"
                )
        #如果返回參數(shù)remove_on_exit，那么刪除該container
            if self.params.get('remove_on_exit'):
                self.stop_container()
                self.remove_container()


def generate_module():
    # NOTE(jeffrey4l): add empty string '' to choices let us use
    # pid_mode: "{{ service.pid_mode | default ('') }}" in yaml
#定義參數(shù)字典，ansible module的api規(guī)范
    argument_spec = dict(
        common_options=dict(required=False, type='dict', default=dict()),
    #action參數(shù)，必須傳遞，類型為str,value值必須在choices的列表
        action=dict(required=True, type='str',
                    choices=['compare_container', 'compare_image',
                             'create_volume', 'get_container_env',
                             'get_container_state', 'pull_image',
                             'recreate_or_restart_container',
                             'remove_container', 'remove_volume',
                             'restart_container', 'start_container',
                             'stop_container']),
        api_version=dict(required=False, type='str', default='auto'),
        auth_email=dict(required=False, type='str'),
        auth_password=dict(required=False, type='str'),
        auth_registry=dict(required=False, type='str'),
        auth_username=dict(required=False, type='str'),
        detach=dict(required=False, type='bool', default=True),
        labels=dict(required=False, type='dict', default=dict()),
        name=dict(required=False, type='str'),
        environment=dict(required=False, type='dict'),
        image=dict(required=False, type='str'),
        ipc_mode=dict(required=False, type='str', choices=['host', '']),
        cap_add=dict(required=False, type='list', default=list()),
        security_opt=dict(required=False, type='list', default=list()),
        pid_mode=dict(required=False, type='str', choices=['host', '']),
        privileged=dict(required=False, type='bool', default=False),
        graceful_timeout=dict(required=False, type='int', default=10),
        remove_on_exit=dict(required=False, type='bool', default=True),
        restart_policy=dict(required=False, type='str', choices=[
                            'no',
                            'never',
                            'on-failure',
                            'always',
                            'unless-stopped']),
        restart_retries=dict(required=False, type='int', default=10),
        tls_verify=dict(required=False, type='bool', default=False),
        tls_cert=dict(required=False, type='str'),
        tls_key=dict(required=False, type='str'),
        tls_cacert=dict(required=False, type='str'),
        volumes=dict(required=False, type='list'),
        volumes_from=dict(required=False, type='list')
    )

# 屬性依賴ansible module的api規(guī)范，如start_container這個action,  #必須要image和name這個兩個屬性。
    required_if = [
        ['action', 'pull_image', ['image']],
        ['action', 'start_container', ['image', 'name']],
        ['action', 'compare_container', ['name']],
        ['action', 'compare_image', ['name']],
        ['action', 'create_volume', ['name']],
        ['action', 'get_container_env', ['name']],
        ['action', 'get_container_state', ['name']],
        ['action', 'recreate_or_restart_container', ['name']],
        ['action', 'remove_container', ['name']],
        ['action', 'remove_volume', ['name']],
        ['action', 'restart_container', ['name']],
        ['action', 'stop_container', ['name']]
    ]
    #實例化
module = AnsibleModule(
        argument_spec=argument_spec,
        required_if=required_if,
        bypass_checks=False
    )
    
#以下部分主要做環(huán)境變量和通用參數(shù)以及特殊參數(shù)的更新。
    new_args = module.params.pop('common_options', dict())

    # NOTE(jeffrey4l): merge the environment
    env = module.params.pop('environment', dict())
    if env:
        new_args['environment'].update(env)

    for key, value in module.params.items():
        if key in new_args and value is None:
            continue
        new_args[key] = value

    # if pid_mode = ""/None/False, remove it
    if not new_args.get('pid_mode', False):
        new_args.pop('pid_mode', None)
    # if ipc_mode = ""/None/False, remove it
    if not new_args.get('ipc_mode', False):
        new_args.pop('ipc_mode', None)

    module.params = new_args
# 返回為AnsibleModule實例
    return module


def main():
    module = generate_module()

    try:
        dw = DockerWorker(module)
        # TODO(inc0): We keep it bool to have ansible deal with consistent
        # types. If we ever add method that will have to return some
        # meaningful data, we need to refactor all methods to return dicts.
        #返回值 result是action傳遞的函數(shù)名的運行成功與否的結(jié)果，意義#不大
    result = bool(getattr(dw, module.params.get('action'))())
        module.exit_json(changed=dw.changed, result=result)
    except Exception:
        module.exit_json(failed=True, changed=True,
                         msg=repr(traceback.format_exc()))

# import module snippets
from ansible.module_utils.basic import *  # noqa
if __name__ == '__main__':
    main()

Kolla_toolbox.py，在toolbox容器中運行ansible命令。這個我就不展開了，只貼關(guān)鍵代碼。

#生成commandline函數(shù)，只包含ansible的命令
def gen_commandline(params):
    command = ['ansible', 'localhost']
    ....
    ....
    return command

#主函數(shù)
def main():
    ....
....
    client = get_docker_client()(
        version=module.params.get('api_version'))
#調(diào)用函數(shù)，獲取命令（dict類型）
    command_line = gen_commandline(module.params)
#過濾名稱為kolla_toolbox的容器列表
    kolla_toolbox = client.containers(filters=dict(name='kolla_toolbox',
                                                   status='running'))
    if not kolla_toolbox:
        module.fail_json(msg='kolla_toolbox container is not running.')
    #默認只有一個，所有選了數(shù)組的第一個。kolla_toolbox變量名不建議重復(fù)使用，開源代碼就是坑。
    kolla_toolbox = kolla_toolbox[0]
#在容器中執(zhí)行命令
    job = client.exec_create(kolla_toolbox, command_line)
    output = client.exec_start(job)
....
....
    module.exit_json(**ret)

****Kolla_container_facts.py**** 調(diào)用dockerclient的python接口獲取指定容器的facts信息，只傳遞一個name值即可。result類型是dict(changed=xxx, _containers=[]),代碼不展開了。

Ansible/action_plugins/

該目錄下記錄了自定義的action的plugins，這些plugins在master上運行。但也可以在library目錄下定義同名空文件，可以當作module使用。這里有兩個代碼文件，merge_configs.py和merge_yaml.py，用于conf和yml配置文件的合并。這里就分下下merge_yaml.py這個action plugin.
Merge_yaml.py，在task的參數(shù)sources中傳遞多個yml文件，合并之后輸出到目標節(jié)點的dest中。期間在合并的同時，進行了參數(shù)模擬變量的渲染工作，最后調(diào)用copy模塊把渲染后的數(shù)據(jù)文件復(fù)制過去。分析代碼如下。
from ansible.plugins import action

#繼承父類action.ActionBase
class ActionModule(action.ActionBase):

    TRANSFERS_FILES = True

    def read_config(self, source):
        result = None
        # Only use config if present
        if os.access(source, os.R_OK):
            with open(source, 'r') as f:
                template_data = f.read()
            # 渲染template模板數(shù)據(jù)，因為最終執(zhí)行copy模塊的時候，
            # 變量被重新還原了，所以這里要先template變量先渲染，
            # 因為有些變量對可能在copy模塊中會消失
            template_data = self._templar.template(template_data)
            #把YAML數(shù)據(jù)，轉(zhuǎn)化為dict對象
            result = safe_load(template_data)
        return result or {}

    # 自定義action plugin必須實現(xiàn)的方法
    def run(self, tmp=None, task_vars=None):
        #task_vars是這個task的一些外傳入的變量,
        # 如host vars, group vars, config vars,etc
        if task_vars is None:
            task_vars = dict()
        #自定義action plugin必須調(diào)用父類的run方法
        result = super(ActionModule, self).run(tmp, task_vars)

        # NOTE(jeffrey4l): Ansible 2.1 add a remote_user param to the
        # _make_tmp_path function.  inspect the number of the args here. In
        # this way, ansible 2.0 and ansible 2.1 are both supported
        #創(chuàng)建tmp臨時目錄，兼容2.0以后的版本
        make_tmp_path_args = inspect.getargspec(self._make_tmp_path)[0]
        if not tmp and len(make_tmp_path_args) == 1:
            tmp = self._make_tmp_path()
        if not tmp and len(make_tmp_path_args) == 2:
            remote_user = (task_vars.get('ansible_user')
                           or self._play_context.remote_user)
            tmp = self._make_tmp_path(remote_user)
        # save template args.
        # _task.args是這個task的參數(shù)，這里把參數(shù)中key為vars的對應(yīng)值
        # 保存到extra_vars變量中
        extra_vars = self._task.args.get('vars', list())
        # 備份template的可用變量
        old_vars = self._templar._available_variables
        # 將task_vars和extra_vars的所有變量merge到一起，賦值到temp_vars
        temp_vars = task_vars.copy()
        temp_vars.update(extra_vars)
        #把最新的變量數(shù)據(jù)設(shè)置到templar對象（模板對象）
        self._templar.set_available_variables(temp_vars)

        output = {}
        # 獲取task的參數(shù)為sources的values值,可能是單個文件，
        # 也有可能是多個文件組成的list
        sources = self._task.args.get('sources', None)
        #非數(shù)組，轉(zhuǎn)化為只有一個item的數(shù)組
        if not isinstance(sources, list):
            sources = [sources]
        #便歷sources數(shù)組，讀取文件中內(nèi)容，并合并更新
        #dict.update方式有去重效果，相當于merge
        for source in sources:
            output.update(self.read_config(source))

        # restore original vars
        #還原templar對象的變量
        self._templar.set_available_variables(old_vars)
        #把最新的合并好的數(shù)據(jù)output傳遞到遠端的target host。復(fù)制給xfered變量
        remote_path = self._connection._shell.join_path(tmp, 'src')
        xfered = self._transfer_data(remote_path,
                                     dump(output,
                                          default_flow_style=False))
        #把本task的參數(shù)拷貝，作為新模塊的參數(shù)new_module_args
        new_module_args = self._task.args.copy()
        #更新new_module_args的src的值,后面copy模塊的參數(shù)要求
        new_module_args.update(
            dict(
                src=xfered
            )
        )
        #刪除new_module_args的sources參數(shù)，后面copy模塊的參數(shù)要求
        del new_module_args['sources']
        #傳入最新的參數(shù)new_module_args，task_vars執(zhí)行copy的module
        result.update(self._execute_module(module_name='copy',
                                           module_args=new_module_args,
                                           task_vars=task_vars,
                                           tmp=tmp))
        #返回result， action plugin 接口的要求
        return result

Ansible/inventory/all-in-one

#control主機組包含本地localhost節(jié)點，連接方式為local
[control]
localhost       ansible_connection=local

[network]
localhost       ansible_connection=local

#neutron主機組包含network組下的所有節(jié)點
[neutron:children]
network

# Neutron
#neutron-server主機組包含control組下的所有節(jié)點
[neutron-server:children]
control

#neutron-dhcp-agent主機組包含neutron組下的所有節(jié)點
[neutron-dhcp-agent:children]
neutron

[neutron-l3-agent:children]
neutron

[neutron-lbaas-agent:children]
neutron

[neutron-metadata-agent:children]
neutron

[neutron-vpnaas-agent:children]
neutron

[neutron-bgp-dragent:children]
neutron

Ansible/site.yml

#調(diào)用ansible的setup獲取節(jié)點的facts。gather_facts被設(shè)置為false是為了避免ansible再次去gathering facts.
- name: Gather facts for all hosts
  hosts: all
  serial: '{{ serial|default("0") }}'
  gather_facts: false
  tasks:
    - setup:
  tags: always

# NOTE(pbourke): This case covers deploying subsets of hosts using --limit. The
# limit arg will cause the first play to gather facts only about that node,
# meaning facts such as IP addresses for rabbitmq nodes etc. will be undefined
# in the case of adding a single compute node.
# We don't want to add the delegate parameters to the above play as it will
# result in ((num_nodes-1)^2) number of SSHs when running for all nodes
# which can be very inefficient.

- name: Gather facts for all hosts (if using --limit)
  hosts: all
  serial: '{{ serial|default("0") }}'
  gather_facts: false
  tasks:
    - setup:
      delegate_facts: True
      delegate_to: "{{ item }}"
      with_items: "{{ groups['all'] }}"
      when:
        - (play_hosts | length) != (groups['all'] | length)

#檢測openstack_release全局變量信息，默認在globals.yml是不配置的，而
#在ansible/group_vars/all.yml中配置的默認值是auto。這里的兩個tasks
#就是如果在auto的場景下，通過python的pbr包去檢測安裝好的#kolla-ansible版本，再將該版本號賦值給openstack_release變量。
#這里用到了ansible自帶的local_action模塊和register中間信息存儲模塊。
- name: Detect openstack_release variable
  hosts: all
  gather_facts: false
  tasks:
    - name: Get current kolla-ansible version number
      local_action: command python -c "import pbr.version; print(pbr.version.VersionInfo('kolla-ansible'))"
      register: kolla_ansible_version
      changed_when: false
      when: openstack_release == "auto"

    - name: Set openstack_release variable
      set_fact:
        openstack_release: "{{ kolla_ansible_version.stdout }}"
      when: openstack_release == "auto"
  tags: always

#對所有節(jié)點進行recheck檢查，前提條件是ansible-playbook命令傳遞的#action的值是precheck。
- name: Apply role prechecks
  gather_facts: false
  hosts:
    - all
  roles:
    - role: prechecks
      when: action == "precheck"

# 基于ntp時間同步角色的部署，hosts組為chrony-server和chrony
#前提條件是enable_chrony變量是否是yes，該值可在etc/kolla/globals.yml
#中配置，默認是no。
- name: Apply role chrony
  gather_facts: false
  hosts:
    - chrony-server
    - chrony
  serial: '{{ serial|default("0") }}'
  roles:
    - { role: chrony,
        tags: chrony,
        when: enable_chrony | bool }

#部署neutron角色，這里部署的節(jié)點除了neutron相關(guān)的host組之外，還包括#compute和manila-share（openstack的一個文件共享組件）組。
- name: Apply role neutron
  gather_facts: false
  hosts:
    - neutron-server
    - neutron-dhcp-agent
    - neutron-l3-agent
    - neutron-lbaas-agent
    - neutron-metadata-agent
    - neutron-vpnaas-agent
    - compute
    - manila-share
  serial: '{{ serial|default("0") }}'
  roles:
    - { role: neutron,
        tags: neutron,
        when: enable_neutron | bool }

Ansible/role/neutron/task 檢查場景

該場景的action是precheck。由tasks/main.yml引用precheck.yml。

---
# kolla_container_facts是自定義的library，上文已經(jīng)分析過代碼，
# 用于獲取容器名為neutron_server的一些容器屬性數(shù)據(jù)，注冊到中間變量container_facts
- name: Get container facts
  kolla_container_facts:
    name:
      - neutron_server
  register: container_facts

# 中間變量container_facts沒有找到neutron_server關(guān)鍵字且該主機在neutron-server主機組中，
# 判斷neutron_server_port 端口是否已經(jīng)stopped
- name: Checking free port for Neutron Server
  wait_for:
    host: "{{ hostvars[inventory_hostname]['ansible_' + api_interface]['ipv4']['address'] }}"
    port: "{{ neutron_server_port }}"
    connect_timeout: 1
    timeout: 1
    state: stopped
  when:
    - container_facts['neutron_server'] is not defined
    - inventory_hostname in groups['neutron-server']

# enable_neutron_agent_ha是true,且只規(guī)劃了多個一個dhcp和l3服務(wù)節(jié)點，給出fail提示
- name: Checking number of network agents
  local_action: fail msg="Number of network agents are less than two when enabling agent ha"
  changed_when: false
  when:
    - enable_neutron_agent_ha | bool
    - groups['neutron-dhcp-agent'] | length < 2
      or groups['neutron-l3-agent'] | length < 2

# When MountFlags is set to shared, a signal bit configured on 20th bit of a number
# We need to check the 20th bit. 2^20 = 1048576. So we are validating against it.
# 檢查docker服務(wù)的MountFlags是否設(shè)置為了shared
- name: Checking if 'MountFlags' for docker service is set to 'shared'
  command: systemctl show docker
  register: result
  changed_when: false
  failed_when: result.stdout.find('MountFlags=1048576') == -1
  when:
    - (inventory_hostname in groups['neutron-dhcp-agent']
       or inventory_hostname in groups['neutron-l3-agent']
       or inventory_hostname in groups['neutron-metadata-agent'])
    - ansible_os_family == 'RedHat' or ansible_distribution == 'Ubuntu'

Ansible/role/neutron/task 部署場景

該場景的action是deploy。由tasks/main.yml引用deploy.yml

# enforce ironic usage only with openvswitch
# 裸機部署檢查，檢查ironic服務(wù)必須啟動，neutron的plugin必須使用OpenvSwitch

- include: ironic-check.yml

#在neutron-server的節(jié)點執(zhí)行注冊
- include: register.yml
  when: inventory_hostname in groups['neutron-server']

#執(zhí)行配置，拷貝配置文件，啟動組件容器主要都在這里實現(xiàn)
- include: config.yml

#在nova fake driver模擬場景下，計算節(jié)點執(zhí)行config-neutron-fake.yml，不詳細分析
#nova fake driver可以在單個計算節(jié)點中創(chuàng)建多個docker容器運行novc-compute，
#Nova fake driver can not work with all-in-one deployment. This is because the fake
#neutron-openvswitch-agent for the fake nova-compute container conflicts with
#neutron-openvswitch-agent on the compute nodes. Therefore, in the inventory
#the network node must be different than the compute node.
- include: config-neutron-fake.yml
  when:
    - enable_nova_fake | bool
    - inventory_hostname in groups['compute']

#在neutron-server的節(jié)點執(zhí)行創(chuàng)建數(shù)據(jù)庫，創(chuàng)建容器
#bootstrap.yml會去創(chuàng)建數(shù)據(jù)庫相關(guān)信息，結(jié)束后會去調(diào)用#bootstrap_servcie.yml該文件是用于在server節(jié)點上創(chuàng)建容器。
- include: bootstrap.yml
  when: inventory_hostname in groups['neutron-server']

#執(zhí)行handlers目錄下的task任務(wù)
- name: Flush Handlers
  meta: flush_handlers

Register.yml, 往keystone中注冊neutron服務(wù)的鑒權(quán)相關(guān)信息。

---
# 在keystone創(chuàng)建neutron的service 和endpoint
# kolla_toolbox見library分析，用于在toolbox容器中執(zhí)行ansible命令
# kolla_keystone_service模塊是kolla-ansible的父項目kolla中的代碼，已經(jīng)是一個可調(diào)用的ansible模塊
#service名稱為neutron,對應(yīng)的endpoint分為內(nèi)部，管理員，公共三個。
# 變量主要在ansible/role/neutron/defauts/main.yml和ansible/group_vars/all.yml中
- name: Creating the Neutron service and endpoint
  kolla_toolbox:
    module_name: "kolla_keystone_service"
    module_args:
      service_name: "neutron"
      service_type: "network"
      description: "Openstack Networking"
      endpoint_region: "{{ openstack_region_name }}"
      url: "{{ item.url }}"
      interface: "{{ item.interface }}"
      region_name: "{{ openstack_region_name }}"
      auth: "{{ '{{ openstack_neutron_auth }}' }}"
    module_extra_vars:
      openstack_neutron_auth: "{{ openstack_neutron_auth }}"
  run_once: True
  with_items:
    - {'interface': 'admin', 'url': '{{ neutron_admin_endpoint }}'}
    - {'interface': 'internal', 'url': '{{ neutron_internal_endpoint }}'}
    - {'interface': 'public', 'url': '{{ neutron_public_endpoint }}'}
    
# 同上，創(chuàng)建項目，用戶，角色。經(jīng)分析openstack_neutron_auth變量實際為#openstack的admin的auth。
- name: Creating the Neutron project, user, and role
  kolla_toolbox:
    module_name: "kolla_keystone_user"
    module_args:
      project: "service"
      user: "{{ neutron_keystone_user }}"
      password: "{{ neutron_keystone_password }}"
      role: "admin"
      region_name: "{{ openstack_region_name }}"
      auth: "{{ '{{ openstack_neutron_auth }}' }}"
    module_extra_vars:
      openstack_neutron_auth: "{{ openstack_neutron_auth }}"
  run_once: True

Config.yml，配置文件合并下發(fā)，創(chuàng)建或重啟容器。

#調(diào)用sysctl模塊，配置ip轉(zhuǎn)發(fā)相關(guān)配置
- name: Setting sysctl values
  vars:
    neutron_l3_agent: "{{ neutron_services['neutron-l3-agent'] }}"
    neutron_vpnaas_agent: "{{ neutron_services['neutron-vpnaas-agent'] }}"
  sysctl: name={{ item.name }} value={{ item.value }} sysctl_set=yes
  with_items:
    - { name: "net.ipv4.ip_forward", value: 1}
    - { name: "net.ipv4.conf.all.rp_filter", value: 0}
    - { name: "net.ipv4.conf.default.rp_filter", value: 0}
  when:
    - set_sysctl | bool
    - (neutron_l3_agent.enabled | bool and neutron_l3_agent.host_in_groups | bool)
      or (neutron_vpnaas_agent.enabled | bool and  neutron_vpnaas_agent.host_in_groups | bool)

# 創(chuàng)建neutron各服務(wù)的配置文件目錄，前提條件主要看host_in_groups變量，這個在
# ansible/role/neutron/defauts/main.yml文件中進行了詳細的定義
- name: Ensuring config directories exist
  file:
    path: "{{ node_config_directory }}/{{ item.key }}"
    state: "directory"
    recurse: yes
  when:
    - item.value.enabled | bool
    - item.value.host_in_groups | bool
  with_dict: "{{ neutron_services }}"

....
....

#下發(fā)配置文件到指定目錄，三份陪配置文件合一，merge_conifgs模塊在action plugin中已經(jīng)分析過了
#文件下發(fā)完了之后，通知相應(yīng)組件的容器重啟。在handlers目錄下。
# 重啟容器這個操作會調(diào)用recreate_or_restart_container這個action，第一次會創(chuàng)建容器。
- name: Copying over neutron_lbaas.conf
  vars:
    service_name: "{{ item.key }}"
    services_need_neutron_lbaas_conf:
      - "neutron-server"
      - "neutron-lbaas-agent"
  merge_configs:
    sources:
      - "{{ role_path }}/templates/neutron_lbaas.conf.j2"
      - "{{ node_custom_config }}/neutron/neutron_lbaas.conf"
      - "{{ node_custom_config }}/neutron/{{ inventory_hostname }}/neutron_lbaas.conf"
    dest: "{{ node_config_directory }}/{{ item.key }}/neutron_lbaas.conf"
  register: neutron_lbaas_confs
  when:
    - item.value.enabled | bool
    - item.value.host_in_groups | bool
    - item.key in services_need_neutron_lbaas_conf
  with_dict: "{{ neutron_services }}"
  notify:
    - "Restart {{ item.key }} container"

....
....
#kolla_docker是自定義的模塊，通過調(diào)用compare_container查找該節(jié)點的所有neutron service容器
- name: Check neutron containers
  kolla_docker:
    action: "compare_container"
    common_options: "{{ docker_common_options }}"
    name: "{{ item.value.container_name }}"
    image: "{{ item.value.image }}"
    privileged: "{{ item.value.privileged | default(False) }}"
    volumes: "{{ item.value.volumes }}"
  register: check_neutron_containers
  when:
    - action != "config"
    - item.value.enabled | bool
    - item.value.host_in_groups | bool
  with_dict: "{{ neutron_services }}"
  notify:
    - "Restart {{ item.key }} container"

Bootstrap.yml，創(chuàng)建neutron數(shù)據(jù)庫對象。
---
# kolla_toolbox自定義模塊，在toolbox容器中調(diào)用mysql_db的ansible模塊，創(chuàng)建db
# delegate_to指定在第一個neutron-server上執(zhí)行，run_onece只運行一次
- name: Creating Neutron database
kolla_toolbox:
module_name: mysql_db
module_args:
login_host: "{{ database_address }}"
login_port: "{{ database_port }}"
login_user: "{{ database_user }}"
login_password: "{{ database_password }}"
name: "{{ neutron_database_name }}"
register: database
run_once: True
delegate_to: "{{ groups['neutron-server'][0] }}"

# 創(chuàng)建neuron數(shù)據(jù)庫的用戶并設(shè)置權(quán)限
- name: Creating Neutron database user and setting permissions
  kolla_toolbox:
    module_name: mysql_user
    module_args:
      login_host: "{{ database_address }}"
      login_port: "{{ database_port }}"
      login_user: "{{ database_user }}"
      login_password: "{{ database_password }}"
      name: "{{ neutron_database_name }}"
      password: "{{ neutron_database_password }}"
      host: "%"
      priv: "{{ neutron_database_name }}.*:ALL"
      append_privs: "yes"
  run_once: True
  delegate_to: "{{ groups['neutron-server'][0] }}"

#數(shù)據(jù)庫改變之后，調(diào)用bootstrap_service.yml
- include: bootstrap_service.yml
  when: database.changed

Bootstrap_service.yml，創(chuàng)建bootrsap_neutron，bootrsap_neutron_lbassd-agent，bootrsap_neutron_vpnaas_agent容器。

Ansible/role/neutron/handlers/main.yml，重建或重啟neutron相關(guān)容器。列出一個分析下。

# 舉例分析：neutron_lbaas_confs變量是之前執(zhí)行的時候注冊的變量，這里的task變量
# neutron_lbaas_conf從neutron_lbaas_confs的結(jié)果中取值分析作為when的條件判斷
# kolla_docker模塊的volumes參數(shù)是從role/neutron/defaults/main.yml中獲取，如下，
#volumes:
#      - "{{ node_config_directory }}/neutron-lbaas-agent/:{{
#             container_config_directory }}/:ro"
#      - "/etc/localtime:/etc/localtime:ro"
#      - "/run:/run:shared"
#      - "kolla_logs:/var/log/kolla/"
#且在config.yml已經(jīng)把neutron_lbaas的配置文件下發(fā){{ node_config_directory }}/
# neutron-lbaas-agent/這個目錄系了。 容器中的路徑container_config_directory變量
# 可以在groups_var中找到,值為 /var/lib/kolla/config_files。
- name: Restart neutron-server container
  vars:
    service_name: "neutron-server"
    service: "{{ neutron_services[service_name] }}"
    config_json: "{{ neutron_config_jsons.results|selectattr('item.key', 'equalto', service_name)|first }}"
    neutron_conf: "{{ neutron_confs.results|selectattr('item.key', 'equalto', service_name)|first }}"
    neutron_lbaas_conf: "{{ neutron_lbaas_confs.results|selectattr('item.key', 'equalto', service_name)|first }}"
    neutron_ml2_conf: "{{ neutron_ml2_confs.results|selectattr('item.key', 'equalto', service_name)|first }}"
    policy_json: "{{ policy_jsons.results|selectattr('item.key', 'equalto', service_name)|first }}"
    neutron_server_container: "{{ check_neutron_containers.results|selectattr('item.key', 'equalto', service_name)|first }}"
  kolla_docker:
    action: "recreate_or_restart_container"
    common_options: "{{ docker_common_options }}"
    name: "{{ service.container_name }}"
    image: "{{ service.image }}"
    volumes: "{{ service.volumes }}"
    privileged: "{{ service.privileged | default(False) }}"
  when:
    - action != "config"
    - service.enabled | bool
    - service.host_in_groups | bool
    - config_json | changed
      or neutron_conf | changed
      or neutron_lbaas_conf | changed
      or neutron_vpnaas_conf | changed
      or neutron_ml2_conf | changed
      or policy_json | changed
      or neutron_server_container | changed

Ansible/role/neutron/task 下拉鏡像

Pull.yml 下拉鏡像
#調(diào)用kolla_docker的pull_image 下拉鏡像
- name: Pulling neutron images
kolla_docker:
action: "pull_image"
common_options: "{{ docker_common_options }}"
image: "{{ item.value.image }}"
when:
- item.value.enabled | bool
- item.value.host_in_groups | bool
with_dict: "{{ neutron_services }}"

流程介紹

鏡像路徑

kolla-ansible\ansible\roles\chrony\defaults\main.yml文件中的一段，如下:

#docker_registry是本地注冊的registry地址，在global.yml中會配置。Registry虛擬機為會從定向5000端口帶宿主機的4000端口。
docker_namespace也在global.yml定義，如果從官網(wǎng)下載源碼鏡像的話，配置成lokolla。
kolla_base_distro默認centos，也可以配置成ubuntu
kolla_install_type 默認為binary, 我們配置在global.yml配置成了source
例子：
docker_registry：192.168.102.15:4000
docker_namespace：lokolla
kolla_base_distro：“centos”
kolla_install_type: source
openstack_release： auto (自發(fā)現(xiàn)，前文有講到)
所以最后的chrony_image_full為192.168.102.15:4000/lokolla/centos-source-chrony:4.0.2

chrony_image: "{{ docker_registry ~ '/' if docker_registry else '' }}{{ docker_namespace }}/{{ kolla_base_distro }}-{{ kolla_install_type }}-chrony"
chrony_tag: "{{ openstack_release }}"
chrony_image_full: "{{ chrony_image }}:{{ chrony_tag }}"