實現容器memory限制

在構造容器01-實現run命令的基礎上需要做以下改動.

1. 修改run命令

加入參數-m表示接受memory限制(command/command.go)

var RunCommand = cli.Command{
    Name: "run",
    Flags: []cli.Flag {
        cli.BoolFlag{
            Name: "it",
            Usage: "enable tty",
        },
        cli.StringFlag{
            Name: "m",
            Usage: "limit memory usage",
        },
    },
    Action: func(c *cli.Context) error {
        tty     := c.Bool("it")
        memory  := c.String("m")
        command := c.Args().Get(0)
        Run(command, tty, memory)
        return nil
    },
}

2. 實現一些utils函數

當前結構如下所示

root@nicktming:~/go/src/github.com/nicktming# tree mydocker
mydocker
|-- cgroups
|   |-- cgroup-manager.go
|   |-- subsystems
|   |   |-- memory.go
|   |   `-- utils.go
|   `-- utils_test.go
|-- command
|   |-- command.go
|   |-- init.go
|   `-- run.go
|-- main.go
|-- README.md
|-- test
|   `-- syscall
|       `-- TestExec.go
`-- urfave-cli-examples
    |-- test01.go
    |-- test02.go
    `-- test03.go

2.1 實現找到對應subsystem的目錄位置

根據subsystem的類型找到對應的hierarchy, 從而可以在該hierarchy創建子cgroup, 進而把進程添加到此cgroup的限制中, 從而達到在此subsystem上限制進程的作用. 在cgroups/utils-test中.

func Test000(t *testing.T)  {
    mountPath := FindCgroupMountPoint("memory")
    log.Printf("mountPath:%s\n", mountPath)
}

cgroups/subsystems/utils.go如下:

func FindCgroupMountPoint(subsystem string) string {
    f, err := os.Open("/proc/self/mountinfo")
    if err != nil {
        log.Printf("Error open file error : %v\n", err)
        return ""
    }
    defer f.Close()

    bfRd := bufio.NewReader(f)
    for {
        line, err := bfRd.ReadBytes('\n')
        if err != nil {
            if err == io.EOF {
                return ""
            }
        }
        parts := strings.Split(string(line), " ")
        if strings.Contains(parts[len(parts) - 1], subsystem) {
            return parts[4]
        }
    }
}

運行結果如下:

root@nicktming:~/go/src/github.com/nicktming/mydocker/cgroups# go test -v utils_test.go -test.run Test000
=== RUN   Test000
2019/03/31 19:21:46 mountPath:/sys/fs/cgroup/memory
--- PASS: Test000 (0.00s)
PASS
ok      command-line-arguments  0.002s

2.2 找到當前容器所在subsystem的hierarchy的絕對路徑

根據subsystem需要找到當前容器的cgroup位置,這樣才可以往里面加入相關的限制. 在cgroups/subsystems/utils-test中.
這是當前路徑memory的文件.

root@nicktming:/sys/fs/cgroup/memory# ls
cgroup.clone_children  memory.kmem.failcnt             memory.kmem.tcp.max_usage_in_bytes  memory.numa_stat            memory.usage_in_bytes
cgroup.event_control   memory.kmem.limit_in_bytes      memory.kmem.tcp.usage_in_bytes      memory.oom_control          memory.use_hierarchy
cgroup.procs           memory.kmem.max_usage_in_bytes  memory.kmem.usage_in_bytes          memory.pressure_level       notify_on_release
cgroup.sane_behavior   memory.kmem.slabinfo            memory.limit_in_bytes               memory.soft_limit_in_bytes  release_agent
memory.failcnt         memory.kmem.tcp.failcnt         memory.max_usage_in_bytes           memory.stat                 tasks
memory.force_empty     memory.kmem.tcp.limit_in_bytes  memory.move_charge_at_immigrate     memory.swappiness           test-limit-memory

root@nicktming:~/go/src/github.com/nicktming# cat mydocker/cgroups/cgroup-manager.go 
package cgroups
const (
    ResourceName = "mydocker"
)

代碼(cgroups/subsystems/utils-test.go)如下:

func FindAbsolutePath(subsystem string) string {
    path := FindCgroupMountPoint(subsystem)
    if path != "" {
        absolutePath := path + "/" + cgroups.ResourceName
        exist, err := PathExists(absolutePath)
        if err != nil {
            log.Printf("PathExists error : %v\n", err)
            return ""
        }
        if !exist {
            err := os.Mkdir(absolutePath, os.ModePerm)
            if err != nil {
                log.Printf("Mkdir absolutePath:%s error : %v\n", err)
                return ""
            }
        }
        return absolutePath
    }
    return ""
}

func PathExists(path string) (bool, error) {
    _, err := os.Stat(path)
    if err == nil {
        return true, nil
    }
    if os.IsNotExist(err) {
        return false, nil
    }
    return false, err
}

func FindCgroupMountPoint(subsystem string) string {
    f, err := os.Open("/proc/self/mountinfo")
    if err != nil {
        log.Printf("Error open file error : %v\n", err)
        return ""
    }
    defer f.Close()

    bfRd := bufio.NewReader(f)
    for {
        line, err := bfRd.ReadBytes('\n')
        if err != nil {
            if err == io.EOF {
                return ""
            }
        }
        parts := strings.Split(string(line), " ")
        if strings.Contains(parts[len(parts) - 1], subsystem) {
            return parts[4]
        }
    }
}

cgroups/utils_test.go如下:

func Test001(t *testing.T)  {
    absolutePath := subsystems.FindAbsolutePath("memory")
    log.Printf("absolutePath:%s\n", absolutePath)
}

運行結果如下:

root@nicktming:~/go/src/github.com/nicktming/mydocker/cgroups# go test -v utils_test.go -test.run Test001
=== RUN   Test001
2019/03/31 19:34:13 absolutePath:/sys/fs/cgroup/memory/mydocker
--- PASS: Test001 (0.00s)
PASS
ok      command-line-arguments  0.002s
// 可以看下是否已經生成
root@nicktming:/sys/fs/cgroup/memory# ls
cgroup.clone_children  memory.kmem.failcnt             memory.kmem.tcp.max_usage_in_bytes  memory.numa_stat            memory.usage_in_bytes  test-limit-memory
cgroup.event_control   memory.kmem.limit_in_bytes      memory.kmem.tcp.usage_in_bytes      memory.oom_control          memory.use_hierarchy
cgroup.procs           memory.kmem.max_usage_in_bytes  memory.kmem.usage_in_bytes          memory.pressure_level       mydocker
cgroup.sane_behavior   memory.kmem.slabinfo            memory.limit_in_bytes               memory.soft_limit_in_bytes  notify_on_release
memory.failcnt         memory.kmem.tcp.failcnt         memory.max_usage_in_bytes           memory.stat                 release_agent
memory.force_empty     memory.kmem.tcp.limit_in_bytes  memory.move_charge_at_immigrate     memory.swappiness           tasks

可以看到已經生成了用于限制memory的當前容器的cgroup mydocker

root@nicktming:/sys/fs/cgroup/memory# ls mydocker/
cgroup.clone_children  memory.kmem.failcnt             memory.kmem.tcp.limit_in_bytes      memory.max_usage_in_bytes        memory.soft_limit_in_bytes  notify_on_release
cgroup.event_control   memory.kmem.limit_in_bytes      memory.kmem.tcp.max_usage_in_bytes  memory.move_charge_at_immigrate  memory.stat                 tasks
cgroup.procs           memory.kmem.max_usage_in_bytes  memory.kmem.tcp.usage_in_bytes      memory.numa_stat                 memory.swappiness
memory.failcnt         memory.kmem.slabinfo            memory.kmem.usage_in_bytes          memory.oom_control               memory.usage_in_bytes
memory.force_empty     memory.kmem.tcp.failcnt         memory.limit_in_bytes               memory.pressure_level            memory.use_hierarchy

3. 實現資源限制

從2中已經可以看到生成當前容器關于某個subsystem的cgroup, 所以這部分將會把相關限制比如內存加入進來.

root@nicktming:/sys/fs/cgroup/memory# pwd
/sys/fs/cgroup/memory
root@nicktming:/sys/fs/cgroup/memory# cat mydocker/tasks
root@nicktming:/sys/fs/cgroup/memory# cat mydocker/memory.limit_in_bytes 
18446744073709551615

代碼如下cgroups/subsystems/memory.go

func Set(content string) error {
    absolutePath := ""
    if absolutePath = FindAbsolutePath("memory"); absolutePath == "" {
        log.Printf("ERROR: absoutePath is empty!\n")
        return fmt.Errorf("ERROR: absoutePath is empty!\n")
    }
    if err := ioutil.WriteFile(path.Join(absolutePath, "memory.limit_in_bytes"), []byte(content),0644); err != nil {
        log.Printf("ERROR write content:%s.\n", content)
        return fmt.Errorf("ERROR write content:%s.\n", content)
    }
    return nil
}

func Apply(pid string) error {
    absolutePath := ""
    if absolutePath = FindAbsolutePath("memory"); absolutePath == "" {
        log.Printf("ERROR: absoutePath is empty!\n")
        return fmt.Errorf("ERROR: absoutePath is empty!\n")
    }
    log.Printf("Apply absolutePath:%s, taskPath:%s\n", absolutePath, path.Join(absolutePath, "tasks"))
    if err := ioutil.WriteFile(path.Join(absolutePath, "tasks"), []byte(pid),0644); err != nil {
        log.Printf("ERROR write pid:%s.\n", pid)
        return fmt.Errorf("ERROR write pid:%s.\n", pid)
    } else {
        log.Printf("err : %v\n", err)
    }
    return nil
}

測試如下:

func Test002(t *testing.T)  {
    subsystems.Set("10M")
    pid := os.Getpid()
    log.Printf("current pid : %s\n", strconv.Itoa(pid))
    subsystems.Apply(strconv.Itoa(pid))
    for i := 0; i < 100; i++ {
        time.Sleep(1 * time.Second)
    }
}

執行結果如下:

root@nicktming:~/go/src/github.com/nicktming/mydocker/cgroups# go test -v utils_test.go -test.run Test002
=== RUN   Test002
2019/03/31 19:44:44 current pid : 18781
2019/03/31 19:44:44 Apply absolutePath:/sys/fs/cgroup/memory/mydocker, taskPath:/sys/fs/cgroup/memory/mydocker/tasks
2019/03/31 19:44:44 err : <nil>

打開另外一個terminal

root@nicktming:/sys/fs/cgroup/memory# cat mydocker/tasks 
18781
18785
root@nicktming:/sys/fs/cgroup/memory# cat mydocker/memory.limit_in_bytes 
10485760
// 運行結束后tasks里面已經沒有進程了
root@nicktming:/sys/fs/cgroup/memory# cat mydocker/tasks 
root@nicktming:/sys/fs/cgroup/memory# cat mydocker/memory.limit_in_bytes 
10485760

4.實現容器資源隔離

在command/run.go中加入對memory的限制, 調用Set和Apply方法.

package command

import (
    "github.com/nicktming/mydocker/cgroups/subsystems"
    "log"
    "os"
    "os/exec"
    "strconv"
    "syscall"
)

func Run(command string, tty bool, memory string)  {
    //cmd := exec.Command(command)

    cmd := exec.Command("/proc/self/exe", "init", command)

    cmd.SysProcAttr = &syscall.SysProcAttr{
        Cloneflags: syscall.CLONE_NEWUTS | syscall.CLONE_NEWPID | syscall.CLONE_NEWNS | syscall.CLONE_NEWNET | syscall.CLONE_NEWIPC,
    }

    if tty {
        cmd.Stderr = os.Stderr
        cmd.Stdout = os.Stdout
        cmd.Stdin = os.Stdin
    }
    /**
     *   Start() will not block, so it needs to use Wait()
     *   Run() will block
     */
    if err := cmd.Start(); err != nil {
        log.Printf("Run Start err: %v.\n", err)
        log.Fatal(err)
    }
    log.Printf("222 before process pid:%d, memory:%s\n", cmd.Process.Pid, memory)

    subsystems.Set(memory)
    subsystems.Apply(strconv.Itoa(cmd.Process.Pid))

    cmd.Wait()
}

在test中加入memory.c做為測試.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

#define MB (1024 * 1024)

int main(int argc, char *argv[])
{
    char *p;
    int i = 0;
    while(1) {
        p = (char *)malloc(MB);
        memset(p, 0, MB);
        printf("%dM memory allocated\n", ++i);
        sleep(1);
    }

    return 0;
}

測試使用.

-------------------------------shell 01-----------------------------------
root@nicktming:~/go/src/github.com/nicktming/mydocker# go build .
root@nicktming:~/go/src/github.com/nicktming/mydocker# ./mydocker run -it -m 5M /bin/sh
// 此時可以打開另外一個terminal
-------------------------------shell 02-----------------------------------
root@nicktming:/sys/fs/cgroup/memory# pwd
/sys/fs/cgroup/memory
root@nicktming:/sys/fs/cgroup/memory# cat mydocker/tasks
23829
root@nicktming:/sys/fs/cgroup/memory# cat mydocker/memory.limit_in_bytes 
5242880
-------------------------------shell 01-----------------------------------
2019/03/31 20:32:56 222 before process pid:23829, memory:5M
2019/03/31 20:32:56 Apply absolutePath:/sys/fs/cgroup/memory/mydocker, taskPath:/sys/fs/cgroup/memory/mydocker/tasks
2019/03/31 20:32:56 err : <nil>
# cp /root/memory .
# ls
cgroups  command  main.go  memory  mydocker  README.md  test  urfave-cli-examples
# ./memory
1M memory allocated
2M memory allocated
3M memory allocated
4M memory allocated
Killed

可以看到確實是可以限制住內存的值. 已經基本成功了, 接下來可以加入刪除功能.

5. 實現資源刪除

資源刪除其實在進程結束的時候把限制解除, 其實就把對應的文件夾給刪除.
在cgroups/subsystems/memory.go中加入Remove方法.

func Remove() error {
    absolutePath := ""
    if absolutePath = FindAbsolutePath("memory"); absolutePath == "" {
        log.Printf("ERROR: absoutePath is empty!\n")
        return fmt.Errorf("ERROR: absoutePath is empty!\n")
    }
    if err := os.RemoveAll(absolutePath); err != nil {
        log.Printf("ERROR: remove absolutePath error:%v\n", err)
        return fmt.Errorf("ERROR: remove absolutePath error:%v\n", err)
    }
    return nil
}

在command/run.go中加入Remove方法.

func Run(command string, tty bool, memory string)  {
    //cmd := exec.Command(command)

    cmd := exec.Command("/proc/self/exe", "init", command)

    cmd.SysProcAttr = &syscall.SysProcAttr{
        Cloneflags: syscall.CLONE_NEWUTS | syscall.CLONE_NEWPID | syscall.CLONE_NEWNS | syscall.CLONE_NEWNET | syscall.CLONE_NEWIPC,
    }

    if tty {
        cmd.Stderr = os.Stderr
        cmd.Stdout = os.Stdout
        cmd.Stdin = os.Stdin
    }
    /**
     *   Start() will not block, so it needs to use Wait()
     *   Run() will block
     */
    if err := cmd.Start(); err != nil {
        log.Printf("Run Start err: %v.\n", err)
        log.Fatal(err)
    }
    log.Printf("222 before process pid:%d, memory:%s\n", cmd.Process.Pid, memory)

    subsystems.Set(memory)
    subsystems.Apply(strconv.Itoa(cmd.Process.Pid))
// 修改處
    defer subsystems.Remove()

    cmd.Wait()
}

至此一個簡單的容器資源限制就結束了.

整體如下

root@nicktming:~/go/src/github.com/nicktming# git clone https://github.com/nicktming/mydocker.git
root@nicktming:~/go/src/github.com/nicktming# cd mydocker
root@nicktming:~/go/src/github.com/nicktming/mydocker# git checkout code-3.2.1
root@nicktming:~/go/src/github.com/nicktming/mydocker# ls
cgroups  command  main.go  memory  pictures  README.md  test  urfave-cli-examples
root@nicktming:~/go/src/github.com/nicktming/mydocker# go build .
root@nicktming:~/go/src/github.com/nicktming/mydocker# ./mydocker run -it -m 12M /bin/sh
2019/04/01 00:52:55 222 before process pid:22014, memory:12M
2019/04/01 00:52:55 Apply absolutePath:/sys/fs/cgroup/memory/mydocker, taskPath:/sys/fs/cgroup/memory/mydocker/tasks
2019/04/01 00:52:55 err : <nil>
# ./memory
1M memory allocated
2M memory allocated
3M memory allocated
4M memory allocated
5M memory allocated
6M memory allocated
7M memory allocated
8M memory allocated
9M memory allocated
10M memory allocated
11M memory allocated
Killed

時序圖如下

時序.png

參考

1. http://www.lxweimin.com/p/7790ca1bc8f6
2. 自己動手寫docker.(基本參考此書,加入一些自己的理解,加深對docker的理解)

全部內容

mydocker.png

1. [mydocker]---環境說明
2. [mydocker]---urfave cli 理解
3. [mydocker]---Linux Namespace
4. [mydocker]---Linux Cgroup
5. [mydocker]---構造容器01-實現run命令
6. [mydocker]---構造容器02-實現資源限制01
7. [mydocker]---構造容器02-實現資源限制02
8. [mydocker]---構造容器03-實現增加管道
9. [mydocker]---通過例子理解存儲驅動AUFS
10. [mydocker]---通過例子理解chroot 和 pivot_root
11. [mydocker]---一步步實現使用busybox創建容器
12. [mydocker]---一步步實現使用AUFS包裝busybox
13. [mydocker]---一步步實現volume操作
14. [mydocker]---實現保存鏡像
15. [mydocker]---實現容器的后臺運行
16. [mydocker]---實現查看運行中容器
17. [mydocker]---實現查看容器日志
18. [mydocker]---實現進入容器Namespace
19. [mydocker]---實現停止容器
20. [mydocker]---實現刪除容器
21. [mydocker]---實現容器層隔離
22. [mydocker]---實現通過容器制作鏡像
23. [mydocker]---實現cp操作
24. [mydocker]---實現容器指定環境變量
25. [mydocker]---網際協議IP
26. [mydocker]---網絡虛擬設備veth bridge iptables
27. [mydocker]---docker的四種網絡模型與原理實現(1)
28. [mydocker]---docker的四種網絡模型與原理實現(2)
29. [mydocker]---容器地址分配
30. [mydocker]---網絡net/netlink api 使用解析
31. [mydocker]---網絡實現
32. [mydocker]---網絡實現測試