Recording rule for expensive ratio

precompute SLO error ratio to keep dashboards and alerts fast

groups: - name: slo rules: - record: job:request_errors:ratio5m expr: sum by(job)(rate(http_requests_total{code=~"5.."}[5m])) / sum by(job)(rate(http_requests_total[5m]))
tool
prometheus30 commands
tags
prometheusgitopsobservabilityperformance

More prometheus commands

  • Validate rules and config with promtool
    promtool check rules rules.yml && promtool check config prometheus.yml

    lint before reload; run in CI to catch broken expr/yaml before deploy

  • Instant query from CLI
    promtool query instant http://localhost:9090 'topk(5, sum by(pod)(rate(container_cpu_usage_seconds_total[5m])))'

    run PromQL without a browser; great for scripted top-N investigations

  • Find cardinality offenders in TSDB
    promtool tsdb analyze /prometheus --limit 20

    lists highest-cardinality metrics and label pairs; first stop for memory blowups

  • Query Prometheus via curl
    curl -sG 'http://prometheus:9090/api/v1/query' --data-urlencode 'query=up' | jq .

    Ad-hoc PromQL query from CLI

  • Instant query via promtool
    promtool query instant http://prometheus:9090 'up{job="kubelet"}'

    Run an instant PromQL query using promtool

all 30 prometheus commands