如何將遠程調用，例如Feign/RestTemplate的調用時間，異常信息等指標收集起來。便于報警和展示呢？這里采用Prometheus+Grafana的方式來實現。本文重點講述下指標如何被Prometheus進行收集的。
1. 指標被Prometheus收集
1.1 簡述Prometheus
Prometheus是一個開源的監控系統，它由以下幾個核心組件構成：
- 數據爬蟲：根據配置的時間定期的通過HTTP抓取metrics數據；
- time-series數據庫：存儲所有的metrics數據；
- 簡單的用戶交互接口：可視化、查詢和監控所有的metrics；
SpringBoot使用Micrometer，一個應用的metrics組件，將actuator metrics整合到外部監控系統中。
為了整合Prometheus，需要增加如下依賴：
<!-- Micrometer Prometheus registry -->
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
一旦增加上述依賴，SpringBoot會自動配置一個PrometheusMeterRegistry和CollectorRegistry來收集和輸出格式化的metrics數據，使得Prometheus服務器可以抓取。
所有應用的metrics數據根據一個叫/prometheus的endpoint來設置是否可用。Prometheus服務器可以周期性的抓取這個endpoint來獲取metrics 麥捶可死數據。
1.2 代碼實現
引入依賴：
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
定義PrometheusMeterRegistry的實例類——該類里面會設置屬性。
public class AbstractMetricsInstance {
protected PrometheusMeterRegistry registry;
public AbstractMetricsInstance(PrometheusMeterRegistry registry) {
this.registry = registry;
}
protected void maximumAllowableTags(String meterName, String tagName, int maxCount) {
if (registry == null || StringUtils.isEmpty(meterName) || StringUtils.isEmpty(tagName) || maxCount < 0) {
return;
}
MeterFilter denyFilter = new OnlyOnceLoggingDenyMeterFilter(() ->
String.format("Metrics reached the maximum number of '%s' tags for '%s'.", tagName, meterName));
registry.config().meterFilter(MeterFilter.maximumAllowableTags(meterName, tagName, maxCount, denyFilter));
}
protected String getMatchPatternByUri(String uri) {
return uri;
}
}
子類實現：
public class AbstractHttpMetricsInstance extends AbstractMetricsInstance {
private static final String TAG_RA = "remote";
private static final String TAG_SN = "svc_name";
private static final String TAG_UP = "uri";
private static final String TAG_SC = "status_code";
private static final String TAG_EX = "exception";
private static final String NONE_VAL = "-";
private final String RSP_SS;
private final String REQ_DS;
private final String RSP_EX;
private final int maxAllowUriTags;
public AbstractHttpMetricsInstance(String RSP_SS, String REQ_DS, String RSP_EX,
PrometheusMeterRegistry registry,
int maxAllowUriTags) {
super(registry);
this.RSP_SS = RSP_SS;
this.REQ_DS = REQ_DS;
this.RSP_EX = RSP_EX;
this.maxAllowUriTags = maxAllowUriTags;
initMaxAllowUriTags();
initPercentilesHistogram();
}
private void initPercentilesHistogram() {
if (registry == null) {
return;
}
registry.config().meterFilter(new MeterFilter() {
@Override
public DistributionStatisticConfig configure(final Meter.Id id, final DistributionStatisticConfig config) {
if (Meter.Type.TIMER.equals(id.getType()) && id.getName().startsWith(REQ_DS)) {
return DistributionStatisticConfig.builder().percentilesHistogram(true)
.sla(Duration.ofMillis(25).toNanos(), Duration.ofMillis(50).toNanos(),
Duration.ofMillis(75).toNanos(), Duration.ofMillis(100).toNanos(),
Duration.ofMillis(200).toNanos(), Duration.ofMillis(500).toNanos(),
Duration.ofMillis(750).toNanos(), Duration.ofSeconds(1).toNanos(),
Duration.ofSeconds(2).toNanos())
.minimumExpectedValue(Duration.ofSeconds(5).toNanos())
.maximumExpectedValue(Duration.ofSeconds(5).toNanos())
.build().merge(config);
}
return config;
}
});
}
private void initMaxAllowUriTags() {
maximumAllowableTags(RSP_SS, TAG_UP, maxAllowUriTags);
maximumAllowableTags(REQ_DS, TAG_UP, maxAllowUriTags);
maximumAllowableTags(RSP_EX, TAG_UP, maxAllowUriTags);
}
/**
* The process for metrics of metric_openfeign_request_duration_seconds_bucket
* The process for metrics of metric_openfeign_request_duration_seconds_count
* The process for metrics of metric_openfeign_request_duration_seconds_max
* The process for metrics of metric_openfeign_request_duration_seconds_sum
*
* @param remoteAddr 遠程服務ip地址
* @param serviceName api的eureka name
* @param path uri
* @param durations 請求耗時
*/
public void requestDurationSeconds(final String remoteAddr, final String serviceName, final String path,
final Long durations) {
if (this.registry == null
|| Strings.isNullOrEmpty(remoteAddr)
|| Strings.isNullOrEmpty(path)) {
return;
}
Tags tags = Tags.of(TAG_RA, remoteAddr)
.and(TAG_SN, StringUtils.isNotBlank(serviceName) ? serviceName : NONE_VAL)
.and(TAG_UP, getMatchPatternByUri(path));
Timer.builder(REQ_DS)
.tags(tags)
.register(registry)
.record(durations, TimeUnit.MILLISECONDS);
}
/**
* The process for metrics of metric_openfeign_response_total
*
* @param remoteAddr 遠程服務ip地址
* @param serviceName api的eureka name
* @param path uri
* @param statusCode 返回狀態碼
*/
public void responseStatusCodeCount(final String remoteAddr, final String serviceName, final String path,
final int statusCode) {
if (this.registry == null
|| Strings.isNullOrEmpty(remoteAddr)
|| Strings.isNullOrEmpty(path)) {
return;
}
Tags tags = Tags.of(TAG_RA, remoteAddr)
.and(TAG_SN, StringUtils.isNotBlank(serviceName) ? serviceName : NONE_VAL)
.and(TAG_UP, getMatchPatternByUri(path))
.and(TAG_SC, String.valueOf(statusCode));
registry.counter(RSP_SS, tags).increment();
}
/**
* The process for metrics of metric_openfeign_client_exception_total
*
* @param remoteAddr 遠程服務ip地址
* @param serviceName api的eureka name
* @param path uri
* @param exception 異常
*/
public void requestExceptionCount(final String remoteAddr, final String serviceName, final String path,
final Throwable exception) {
if (this.registry == null
|| Strings.isNullOrEmpty(path)
|| exception == null) {
return;
}
Tags tags = Tags.of(TAG_RA, remoteAddr)
.and(TAG_SN, StringUtils.isNotBlank(serviceName) ? serviceName : NONE_VAL)
.and(TAG_UP, getMatchPatternByUri(path))
.and(exceptionTag(exception));
registry.counter(RSP_EX, tags).increment();
}
private Tags exceptionTag(final Throwable exception) {
String simpleName = exception.getClass().getSimpleName();
return Tags.of(TAG_EX, simpleName.isEmpty() ? exception.getClass().getName() : simpleName);
}
}
openFeign的Metrics類實現：
@Component
public class OpenFeignMetricsInstance extends AbstractHttpMetricsInstance {
private static final String RSP_SS = "metric.openfeign.response";
private static final String REQ_DS = "metric.openfeign.request.duration";
private static final String RSP_EX = "metric.openfeign.client.exception";
public OpenFeignMetricsInstance(PrometheusMeterRegistry registry) {
super(RSP_SS, REQ_DS, RSP_EX, registry, 1000);
}
}
2. 如何監控RestTemplate的信息
當調用成功時、調用出現異常時均調用OpenFeignMetricsInstance類進行打點收集。
@Service
public class RestTemplateMetricsInterceptor implements ClientHttpRequestInterceptor {
@Autowired
private OpenFeignMetricsInstance openFeignMetricsInstance;
@Override
public ClientHttpResponse intercept(HttpRequest request, byte[] body, ClientHttpRequestExecution execution) throws IOException {
ClientHttpResponse response;
URI uri = request.getURI();
String path = uri.getPath();
String serviceName = uri.getHost();
int status = 599;
try {
final Stopwatch stopwatch = Stopwatch.createStarted();
response = execution.execute(request, body);
stopwatch.stop();
//覆蓋錯誤碼
status = response.getRawStatusCode();
openFeignMetricsInstance.requestDurationSeconds(serviceName, serviceName, path, stopwatch.elapsed(TimeUnit.MILLISECONDS));
} catch (Exception e) {
openFeignMetricsInstance.requestExceptionCount(serviceName, serviceName, path, e);
throw e;
} finally {
openFeignMetricsInstance.responseStatusCodeCount(serviceName, serviceName, path, status);
}
return response;
}
}
ClientHttpRequestInterceptor如何設置到RestTemplate可以參考：Spring—RestTemplate設置Interceptor攔截器代碼實現。
3. 如何監控Feign的信息
文章參考——Feign源碼分析—替換(裝飾)底層client完成Feign接口的監控
