我們在使用openshift的時候,經常會出現每個項目一個jenkins,在項目初始化過程我們需要對slave pod做一些配置,需要在部署jenkins后通過頁面修改,現在我們可以通過configmap方式快速自動的更新。
比如: 配置maven pod的存儲,將maven緩存文件保存在持久化存儲上,以加快下次編譯速度。
比如: 配置maven nodejs的時區,使輸出日志時間準確。
比如: 配置slave pod的資源限額,避免編譯過程出現OOM,因為有的應用編譯過程是比較吃內存的。
比如: 配置多個slave pod,要支持jdk7,8,10,nodejs 6,8,10,12 等情況。。。
Kubernetes Plug-in
Kubernetes Plug-in 是一個動態生成 slave pod 的插件,用于運行jenkins job,job完成后銷毀pod。
此插件已經集成在openshift jenkins中,相關默認配置在jenkins初始化過程會自動處理。
slave pod 默認內置了maven 和 nodejs
如果只是變更maven和nodejs的鏡像,我們在openshift project下修改jenkins的template,在env 部分新加兩個環境變量 MAVEN_SLAVE_IMAGE NODEJS_SLAVE_IMAGE
TZ 時區變量是為了日志時間準確
- name: JENKINS_SERVICE_NAME
value: ${JENKINS_SERVICE_NAME}
- name: JNLP_SERVICE_NAME
value: ${JNLP_SERVICE_NAME}
- name: ENABLE_FATAL_ERROR_LOG_FILE
value: ${ENABLE_FATAL_ERROR_LOG_FILE}
- name: MAVEN_SLAVE_IMAGE
value: harbor.test.bocloud.com/openshift/openshift3/jenkins-slave-maven-rhel7:v3.11.141-settings-oraclejdk678-skope
- name: NODEJS_SLAVE_IMAGE
value: harbor.test.bocloud.com/openshift/openshift3/jenkins-agent-nodejs-6-8-10-12-rhel7:v3.11.141
- name: TZ
value: Asia/Shanghai
OpenShift Sync plug-in
如果要快速便捷自定義 slave pod,openshift 提供了 OpenShift Sync plug-in 插件,此插件默認已安裝。
這個插件會在jenkins 啟動時候查找project下的具備以下特性的imagestream或者configmap
- imagestream 具備label role=jenkins-slave
- imagestream tags 具備 annotation role=jenkins-slave
- configmap 具備label role=jenkins-slave
具備相關label 的imagestream 或者具備相關annotation 的imagestream tag,會自動更新到kubernetes plugin 配置中,可以和自帶的maven與nodejs slave pod一樣,直接被jenkins job調用
configmap 方式提供了與kubernetes plugin配置一致的XML格式文件,通過這種方式可以配置kubernetes plugin 支持的所有格式。
kubernetes plugin 支持的參數見插件官方
https://github.com/jenkinsci/kubernetes-plugin
以下為我們在工作中使用到的configmap,配置了資源限額、環境變量、pvc持久化卷
https://raw.githubusercontent.com/cai11745/k8s-ocp-yaml/master/yaml-file/jenkins-agent-pod-template.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: jenkins-agent-pod-template
labels:
role: jenkins-slave
data:
template1: |-
<org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
<inheritFrom></inheritFrom>
<name>maven</name>
<namespace></namespace>
<privileged>false</privileged>
<capOnlyOnAlivePods>false</capOnlyOnAlivePods>
<alwaysPullImage>false</alwaysPullImage>
<instanceCap>2147483647</instanceCap>
<slaveConnectTimeout>100</slaveConnectTimeout>
<idleMinutes>0</idleMinutes>
<activeDeadlineSeconds>0</activeDeadlineSeconds>
<label>maven</label>
<serviceAccount>jenkins</serviceAccount>
<nodeSelector></nodeSelector>
<nodeUsageMode>NORMAL</nodeUsageMode>
<customWorkspaceVolumeEnabled>false</customWorkspaceVolumeEnabled>
<workspaceVolume class="org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.EmptyDirWorkspaceVolume">
<memory>false</memory>
</workspaceVolume>
<volumes>
<org.csanchez.jenkins.plugins.kubernetes.volumes.PersistentVolumeClaim>
<mountPath>/home/jenkins/.m2/repository/</mountPath>
<claimName>maven-m2</claimName>
<readOnly>false</readOnly>
</org.csanchez.jenkins.plugins.kubernetes.volumes.PersistentVolumeClaim>
</volumes>
<containers>
<org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
<name>jnlp</name>
<image>harbor.test.geely.com/openshift/openshift3/jenkins-slave-maven-rhel7:v3.11.141-settings-oraclejdk678-skope</image>
<privileged>false</privileged>
<alwaysPullImage>true</alwaysPullImage>
<workingDir>/tmp</workingDir>
<command></command>
<args>${computer.jnlpmac} ${computer.name}</args>
<ttyEnabled>false</ttyEnabled>
<resourceRequestCpu>1</resourceRequestCpu>
<resourceRequestMemory>2G</resourceRequestMemory>
<resourceLimitCpu>1</resourceLimitCpu>
<resourceLimitMemory>2G</resourceLimitMemory>
<envVars>
<org.csanchez.jenkins.plugins.kubernetes.model.KeyValueEnvVar>
<key>TZ</key>
<value>Asia/Shanghai</value>
</org.csanchez.jenkins.plugins.kubernetes.model.KeyValueEnvVar>
<org.csanchez.jenkins.plugins.kubernetes.model.KeyValueEnvVar>
<key>LC_ALL</key>
<value>en_US.UTF-8</value>
</org.csanchez.jenkins.plugins.kubernetes.model.KeyValueEnvVar>
</envVars>
<ports/>
<livenessProbe>
<execArgs></execArgs>
<timeoutSeconds>0</timeoutSeconds>
<initialDelaySeconds>0</initialDelaySeconds>
<failureThreshold>0</failureThreshold>
<periodSeconds>0</periodSeconds>
<successThreshold>0</successThreshold>
</livenessProbe>
</org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
</containers>
<envVars/>
<annotations/>
<imagePullSecrets/>
<nodeProperties/>
<yaml></yaml>
<podRetention class="org.csanchez.jenkins.plugins.kubernetes.pod.retention.Default"/>
</org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
template2: |-
<org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
<inheritFrom></inheritFrom>
<name>nodejs</name>
<namespace></namespace>
<privileged>false</privileged>
<capOnlyOnAlivePods>false</capOnlyOnAlivePods>
<alwaysPullImage>false</alwaysPullImage>
<instanceCap>2147483647</instanceCap>
<slaveConnectTimeout>100</slaveConnectTimeout>
<idleMinutes>0</idleMinutes>
<activeDeadlineSeconds>0</activeDeadlineSeconds>
<label>nodejs</label>
<serviceAccount>jenkins</serviceAccount>
<nodeSelector></nodeSelector>
<nodeUsageMode>NORMAL</nodeUsageMode>
<customWorkspaceVolumeEnabled>false</customWorkspaceVolumeEnabled>
<workspaceVolume class="org.csanchez.jenkins.plugins.kubernetes.volumes.workspace.EmptyDirWorkspaceVolume">
<memory>false</memory>
</workspaceVolume>
<volumes>
<org.csanchez.jenkins.plugins.kubernetes.volumes.PersistentVolumeClaim>
<mountPath>/home/jenkins/npm-cache</mountPath>
<claimName>npm-cache</claimName>
<readOnly>false</readOnly>
</org.csanchez.jenkins.plugins.kubernetes.volumes.PersistentVolumeClaim>
</volumes>
<containers>
<org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
<name>jnlp</name>
<image>harbor.test.geely.com/openshift/openshift3/jenkins-agent-nodejs-6-8-10-12-rhel7:v3.11.141-geely</image>
<privileged>false</privileged>
<alwaysPullImage>true</alwaysPullImage>
<workingDir>/tmp</workingDir>
<command></command>
<args>${computer.jnlpmac} ${computer.name}</args>
<ttyEnabled>false</ttyEnabled>
<resourceRequestCpu>1</resourceRequestCpu>
<resourceRequestMemory>3G</resourceRequestMemory>
<resourceLimitCpu>1</resourceLimitCpu>
<resourceLimitMemory>3G</resourceLimitMemory>
<envVars>
<org.csanchez.jenkins.plugins.kubernetes.model.KeyValueEnvVar>
<key>TZ</key>
<value>Asia/Shanghai</value>
</org.csanchez.jenkins.plugins.kubernetes.model.KeyValueEnvVar>
</envVars>
<ports/>
<livenessProbe>
<execArgs></execArgs>
<timeoutSeconds>0</timeoutSeconds>
<initialDelaySeconds>0</initialDelaySeconds>
<failureThreshold>0</failureThreshold>
<periodSeconds>0</periodSeconds>
<successThreshold>0</successThreshold>
</livenessProbe>
</org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
</containers>
<envVars/>
<annotations/>
<imagePullSecrets/>
<nodeProperties/>
<yaml></yaml>
<podRetention class="org.csanchez.jenkins.plugins.kubernetes.pod.retention.Default"/>
</org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
如果不知道XML參數怎么寫,可以找一個jenkins,通過頁面配置好slave pod的參數,然后進入jenkins容器,配置的參數保存在這里 /var/lib/jenkins/config.xml
導入configmap,sync 插件會把configmap內容自動同步到jenkins pod template,不需要重啟jenkins
需要注意幾點:
- 原來的maven nodejs配置會被覆蓋,因為configmap中使用了相同的name
- 刪除configmap,會將jenkins中的兩個slave pod 同步刪除
- 不要直接在 jenkins 頁面中修改 slave pod的參數,因為一重啟就會被configmap重置
- 如果后面不想被configmap同步,但是想保留配置,在jenkins頁面把pod template 名稱改掉,就是第一個maven 或者nodejs參數
,然后把configmap 刪了
更多分享見github
https://github.com/cai11745/k8s-ocp-yaml
參考文檔:
openshift 官方手冊
https://docs.openshift.com/container-platform/3.11/using_images/other_images/jenkins.html