之前寫過kubernetes1.8 kube-scheduler源碼閱讀。基于上篇文章的基礎,重新閱讀了kubernetes1.9的kube-scheduler的代碼。本篇文章是kubernetes1.9的kube-scheduler代碼閱讀筆記。不得不說,kubernetes工程師們,每經過短短三個月的版本發(fā)布,代碼都會經過大大的重構。好的工程師是在不斷重構代碼中,認識得到升華,錘煉出來的。
kubernetes tag v.1.9.0
kube-scheduler的代碼路徑為:plugin/
入口程序:plugin/cmd/kube-scheduler/scheduler.go main()
1. kubernetes1.9對入口程序cmd部分進行了重構
2. 首先,調用app.NewSchedulerCommand(),? NewSchedulerCommand()方法在plugin/cmd/kube-scheduler/app/server.go中,NewSchedulerCommand()方法將調用opts.Run();
3. 創(chuàng)建NewSchedulerServer和啟動調度服務;
4. componentconfig.KubeSchedulerConfiguration{}定義了kube-scheduler的參數(shù)信息;
構建調度服務器:plugin/cmd/kube-scheduler/app/server.go NewSchedulerServer()
1. 調用createClients(),創(chuàng)建apiserver客戶端,通過REST方式訪問APIserver提供的API服務,用來watch pod和node,并調用api server bind接口完成node和pod的Bind操作;
2. 調用record.NewBroadcaster(),創(chuàng)建eventBroadcaster對象,recorder記錄config.SchedulerName的事件;
3. 如果設置了leaderElect,調用makeLeaderElectionConfig()進行選舉;
4. 調用informers.NewSharedInformerFactory(),創(chuàng)建sharedInformerFactory對象
5. 調用factory.NewPodInformer(),創(chuàng)建PodInformer對象,PodInformer用于watch/list non-terminal pods并緩存;
啟動程序:plugin/cmd/kube-scheduler/app/server.go Run()
1. 調用s.SchedulerConfig()方法,創(chuàng)建scheduler.Config對象;
2. 在s.SchedulerConfig()方法中,首先,通過factory.NewConfigFactory()創(chuàng)建ConfigFactory對象,ConfigFactory是scheduler.Configurator接口的實現(xiàn);
3. ConfigFactory對象,包括:nodeInformer、podInformer、pvInformer、pvcInformer、rcInformer、rsInformer、statefulsetInformer、serviceInformer、pdbInformer、storageClassInformer,這些是通過s.InformerFactory創(chuàng)建;
4. 調用scheduler.NewFromConfig(schedulerConfig),創(chuàng)建Scheduler對象;
5. 調用s.Broadcaster.StartRecordingToSink(),接收EventBroadcaster發(fā)送的event,輸出到EventSink;
6. 啟動healthz server和metrics server,提供必要的健康檢查和性能量度;
7. 調用informerFactory.start() ,開始運行Informer,進行緩存;
8. 調用sched.Run(),運行調度程序;
SharedInformers模式設計同時用在k8s的"Controller"中,下面是一段關于SharedInformers模式設計的英文介紹:(摘自https://github.com/kubernetes/community/blob/8decfe4/contributors/devel/controllers.md)
Use SharedInformers. SharedInformers provide hooks to receive notifications of adds, updates, and deletes for a particular resource. They also provide convenience functions for accessing shared caches and determining when a cache is primed.
SharedInformers提供勾子機制,接收到特定資源添加、更新、刪除的通知。并提供函數(shù)更新緩存,啟動執(zhí)行。簡而言之kube-scheduler的"informer"負責:watch/list non-terminal pods, 進行緩存,再從podQueue中獲得NextPod,執(zhí)行調度;
調度程序:plugin/pkg/scheduler/scheduler.go?Run()
1. 等待緩存更新完成;
2. 調用sched.scheduleOne,運行調度流程;
調度流程:plugin/pkg/scheduler/scheduler.go scheduleOne()
1. 調用sched.config.NextPod(),從podQueue緩存中獲得一個Pod;
2. 調用sched.schedule(pod),獲得一個suggestedHost,此操作是同步操作;
3. 調用metrics.SchedulingAlgorithmLatency.Observe(metrics.SinceInMicroseconds(start)),記錄算法延遲的量度;
4. 此Pod將標注為assumedPod;此時Pod并沒有被成功調度;
5. 調用sched.bind(),綁定Pod到suggestedHost,此操作是異步操作;
6. 調用metrics.E2eSchedulingLatency.Observe(),記錄調度延遲的量度;
調度邏輯:plugin/pkg/scheduler/scheduler.go Schedule()
1. 根據(jù)調度策略算法確定一個suggestedHost;
調度算法函數(shù):
1. 調度算法函數(shù)支持plugin模式plugin/pkg/scheduler/algorithmprovider/plugins.go,scheduler的commandLine參數(shù)AlgorithmProvider可以指定調度算法函數(shù);默認使用defaultProvider, defaultPredicates(), defaultPriorities() plugin/pkg/scheduler/algorithmprovider/defaults/default.go;同時,scheduler的commandLine參數(shù)PolicyConfigFile,可以加載自定義的調度策略文件。如:openshift中,/etc/origin/master/scheduler.json定義了調度策略文件。(參考信息:https://docs.openshift.com/container-platform/3.6/admin_guide/scheduling/scheduler.html)
2. FitPredicates:k8s.io/kubernetes/plugin/pkg/scheduler/algorithm/predicates
3. PrioritiesFunc:k8s.io/kubernetes/plugin/pkg/scheduler/algorithm/priorities
4. 調度流程plugin/pkg/scheduler/core/generic_scheduler.go schedule(),調度流程圖如下:(摘自kubernetes調度詳解:http://dockone.io/article/2885)
綁定邏輯:plugin/pkg/scheduler/scheduler.go bind()
1. 調用SchedulerCache.FinishBinding(),更新schedulerCache,將assumedPod的狀態(tài)變?yōu)閑xpired;? schedulerCache實現(xiàn)了schedulercache.cache接口;
2. schedulerCache定義在plugin/pkg/scheduler/schedulercache/cache.go;
3. schedulercache.cache定義在plugin/pkg/scheduler/schedulercache/interface.go;
4. 調用metrics.BindingLatency.Observe(),記錄綁定延遲的量度;
5. 調用?sched.config.Recorder.Eventf(),記錄綁定事件;
相關閱讀:
1.?http://licyhust.com/