Detect dead target with absent

fires when no series match; catches a job that vanished from discovery entirely

absent(up{job="node"} == 1)
tool
prometheus30 commands
tags
prometheusmonitoringtroubleshootingobservability

More prometheus commands

  • Rewrite label with label_replace
    label_replace(node_uname_info, "host", "$1", "nodename", "(.+)")

    extract/derive a new label via regex capture; useful for joins and tidy dashboards

  • Max of per-second rate over window
    max_over_time(rate(node_network_receive_bytes_total[5m])[1h:1m])

    subquery: peak 5m rate sampled each minute over the last hour; spot bursts

  • Aggregate keeping all labels but instance
    sum without(instance) (rate(http_requests_total{code=~"5.."}[5m]))

    without() keeps every label except instance; cleaner than listing many by() labels

  • Recording rule for expensive ratio
    groups: - name: slo rules: - record: job:request_errors:ratio5m expr: sum by(job)(rate(http_requests_total{code=~"5.."}[5m])) / sum by(job)(rate(http_requests_total[5m]))

    precompute SLO error ratio to keep dashboards and alerts fast

  • Validate rules and config with promtool
    promtool check rules rules.yml && promtool check config prometheus.yml

    lint before reload; run in CI to catch broken expr/yaml before deploy

all 30 prometheus commands