Series cardinality per job prometheus
PromQL: count active time series grouped by job label
count by (job) ({__name__=~".+"})
more prometheus
all 30 commands →
Silence an alert
amtool silence add alertname=HighMemory --duration=2h --comment='Investigating'
Check TSDB status
curl -s http://prometheus:9090/api/v1/status/tsdb | jq '{head_chunks: .data.headStats.numChunks, series: .data.headStats.numSeries}'
95th percentile latency from histogram
histogram_quantile(0.95, sum by(le,job)(rate(http_request_duration_seconds_bucket[5m])))
Predict disk full window
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4*3600) < 0
Detect dead target with absent
absent(up{job="node"} == 1)