95th percentile latency from histogram prometheus
p95 latency per job; note: always aggregate _bucket by le before histogram_quantile
histogram_quantile(0.95, sum by(le,job)(rate(http_request_duration_seconds_bucket[5m])))
more prometheus
all 30 commands →
Predict disk full window
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4*3600) < 0
Detect dead target with absent
absent(up{job="node"} == 1)
Rewrite label with label_replace
label_replace(node_uname_info, "host", "$1", "nodename", "(.+)")
Max of per-second rate over window
max_over_time(rate(node_network_receive_bytes_total[5m])[1h:1m])
Aggregate keeping all labels but instance
sum without(instance) (rate(http_requests_total{code=~"5.."}[5m]))