Count series per job

PromQL: count active time series for each job label

count by (job) ({__name__=~".+"})

tool
prometheus30 commands
tags
prometheusmonitoring

More prometheus commands

  • amtool silence add alertname=HighMemory --duration=2h --comment='Investigating'

    Create an Alertmanager silence for 2 hours

  • curl -s http://prometheus:9090/api/v1/status/tsdb | jq '{head_chunks: .data.headStats.numChunks, series: .data.headStats.numSeries}'

    Show TSDB head stats: chunks and active series

  • histogram_quantile(0.95, sum by(le,job)(rate(http_request_duration_seconds_bucket[5m])))

    p95 latency per job. Always aggregate `_bucket` by `le` before histogram_quantile

  • predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4*3600) < 0

    Alert when the trend of the last 6h says the disk will be full within 4h

  • absent(up{job="node"} == 1)

    Fires when no series match, catches a job that disappeared from service discovery

all 30 prometheus commands