Predict when the disk gets full

Alert when the trend of the last 6h says the disk will be full within 4h

predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4*3600) < 0

tool
prometheus30 commands
tags
prometheusdiskmonitoringtroubleshooting

More prometheus commands

  • absent(up{job="node"} == 1)

    Fires when no series match, catches a job that disappeared from service discovery

  • label_replace(node_uname_info, "host", "$1", "nodename", "(.+)")

    Create a new label from a regex capture, useful for joins and cleaner dashboards

  • max_over_time(rate(node_network_receive_bytes_total[5m])[1h:1m])

    Subquery: the highest 5m rate, checked every minute over the last hour, shows traffic bursts

  • sum without(instance) (rate(http_requests_total{code=~"5.."}[5m]))

    without() keeps every label except instance, cleaner than a long by() list

  • groups: - name: slo rules: - record: job:request_errors:ratio5m expr: sum by(job)(rate(http_requests_total{code=~"5.."}[5m])) / sum by(job)(rate(http_requests_total[5m]))

    Precompute the SLO error ratio so dashboards and alerts stay fast

all 30 prometheus commands