序
本文主要研究下flink的checkpoint配置
實(shí)例
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
// start a checkpoint every 1000 ms
env.enableCheckpointing(1000);
// advanced options:
// set mode to exactly-once (this is the default)
env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.EXACTLY_ONCE);
// checkpoints have to complete within one minute, or are discarded
env.getCheckpointConfig().setCheckpointTimeout(60000);
// make sure 500 ms of progress happen between checkpoints
env.getCheckpointConfig().setMinPauseBetweenCheckpoints(500);
// allow only one checkpoint to be in progress at the same time
env.getCheckpointConfig().setMaxConcurrentCheckpoints(1);
// enable externalized checkpoints which are retained after job cancellation
env.getCheckpointConfig().enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
// This determines if a task will be failed if an error occurs in the execution of the task’s checkpoint procedure.
env.getCheckpointConfig().setFailOnCheckpointingErrors(true);
- 使用StreamExecutionEnvironment.enableCheckpointing方法來(lái)設(shè)置開(kāi)啟checkpoint；具體可以使用enableCheckpointing(long interval)，或者enableCheckpointing(long interval, CheckpointingMode mode)；interval用于指定checkpoint的觸發(fā)間隔(
單位milliseconds)，而CheckpointingMode默認(rèn)是CheckpointingMode.EXACTLY_ONCE，也可以指定為CheckpointingMode.AT_LEAST_ONCE - 也可以通過(guò)StreamExecutionEnvironment.getCheckpointConfig().setCheckpointingMode來(lái)設(shè)置CheckpointingMode，一般對(duì)于超低延遲的應(yīng)用(
大概幾毫秒)可以使用CheckpointingMode.AT_LEAST_ONCE，其他大部分應(yīng)用使用CheckpointingMode.EXACTLY_ONCE就可以 - checkpointTimeout用于指定checkpoint執(zhí)行的超時(shí)時(shí)間(
單位milliseconds)，超時(shí)沒(méi)完成就會(huì)被abort掉 - minPauseBetweenCheckpoints用于指定checkpoint coordinator上一個(gè)checkpoint完成之后最小等多久可以出發(fā)另一個(gè)checkpoint，當(dāng)指定這個(gè)參數(shù)時(shí)，maxConcurrentCheckpoints的值為1
- maxConcurrentCheckpoints用于指定運(yùn)行中的checkpoint最多可以有多少個(gè)，用于包裝topology不會(huì)花太多的時(shí)間在checkpoints上面；如果有設(shè)置了minPauseBetweenCheckpoints，則maxConcurrentCheckpoints這個(gè)參數(shù)就不起作用了(
大于1的值不起作用) - enableExternalizedCheckpoints用于開(kāi)啟checkpoints的外部持久化，但是在job失敗的時(shí)候不會(huì)自動(dòng)清理，需要自己手工清理state；ExternalizedCheckpointCleanup用于指定當(dāng)job canceled的時(shí)候externalized checkpoint該如何清理，DELETE_ON_CANCELLATION的話，在job canceled的時(shí)候會(huì)自動(dòng)刪除externalized state，但是如果是FAILED的狀態(tài)則會(huì)保留；RETAIN_ON_CANCELLATION則在job canceled的時(shí)候會(huì)保留externalized checkpoint state
- failOnCheckpointingErrors用于指定在checkpoint發(fā)生異常的時(shí)候，是否應(yīng)該fail該task，默認(rèn)為true，如果設(shè)置為false，則task會(huì)拒絕checkpoint然后繼續(xù)運(yùn)行
flink-conf.yaml相關(guān)配置
#==============================================================================
# Fault tolerance and checkpointing
#==============================================================================
# The backend that will be used to store operator state checkpoints if
# checkpointing is enabled.
#
# Supported backends are 'jobmanager', 'filesystem', 'rocksdb', or the
# <class-name-of-factory>.
#
# state.backend: filesystem
# Directory for checkpoints filesystem, when using any of the default bundled
# state backends.
#
# state.checkpoints.dir: hdfs://namenode-host:port/flink-checkpoints
# Default target directory for savepoints, optional.
#
# state.savepoints.dir: hdfs://namenode-host:port/flink-checkpoints
# Flag to enable/disable incremental checkpoints for backends that
# support incremental checkpoints (like the RocksDB state backend).
#
# state.backend.incremental: false
- state.backend用于指定checkpoint state存儲(chǔ)的backend，默認(rèn)為none
- state.backend.async用于指定backend是否使用異步snapshot(
默認(rèn)為true)，有些不支持async或者只支持async的state backend可能會(huì)忽略這個(gè)參數(shù) - state.backend.fs.memory-threshold，默認(rèn)為1024，用于指定存儲(chǔ)于files的state大小閾值，如果小于該值則會(huì)存儲(chǔ)在root checkpoint metadata file
- state.backend.incremental，默認(rèn)為false，用于指定是否采用增量checkpoint，有些不支持增量checkpoint的backend會(huì)忽略該配置
- state.backend.local-recovery，默認(rèn)為false
- state.checkpoints.dir，默認(rèn)為none，用于指定checkpoint的data files和meta data存儲(chǔ)的目錄，該目錄必須對(duì)所有參與的TaskManagers及JobManagers可見(jiàn)
- state.checkpoints.num-retained，默認(rèn)為1，用于指定保留的已完成的checkpoints個(gè)數(shù)
- state.savepoints.dir，默認(rèn)為none，用于指定savepoints的默認(rèn)目錄
- taskmanager.state.local.root-dirs，默認(rèn)為none
小結(jié)
- 可以通過(guò)使用StreamExecutionEnvironment.enableCheckpointing方法來(lái)設(shè)置開(kāi)啟checkpoint；具體可以使用enableCheckpointing(long interval)，或者enableCheckpointing(long interval, CheckpointingMode mode)
- checkpoint的高級(jí)配置可以配置checkpointTimeout(
用于指定checkpoint執(zhí)行的超時(shí)時(shí)間，單位milliseconds)，minPauseBetweenCheckpoints(用于指定checkpoint coordinator上一個(gè)checkpoint完成之后最小等多久可以出發(fā)另一個(gè)checkpoint)，maxConcurrentCheckpoints(用于指定運(yùn)行中的checkpoint最多可以有多少個(gè)，如果有設(shè)置了minPauseBetweenCheckpoints，則maxConcurrentCheckpoints這個(gè)參數(shù)大于1的值不起作用)，enableExternalizedCheckpoints(用于開(kāi)啟checkpoints的外部持久化，在job failed的時(shí)候externalized checkpoint state無(wú)法自動(dòng)清理，但是在job canceled的時(shí)候可以配置是刪除還是保留state) - 在flink-conf.yaml里頭也有checkpoint的相關(guān)配置，主要是state backend的配置，比如state.backend.async、state.backend.incremental、state.checkpoints.dir、state.savepoints.dir等
