Max of per-second rate over window prometheus
subquery: peak 5m rate sampled each minute over the last hour; spot bursts
max_over_time(rate(node_network_receive_bytes_total[5m])[1h:1m])
more prometheus
all 30 commands →
Aggregate keeping all labels but instance
sum without(instance) (rate(http_requests_total{code=~"5.."}[5m]))
Recording rule for expensive ratio
groups:
- name: slo
rules:
- record: job:request_errors:ratio5m
expr: sum by(job)(rate(http_requests_total{code=~"5.."}[5m])) / sum by(job)(rate(http_requests_total[5m]))
Validate rules and config with promtool
promtool check rules rules.yml && promtool check config prometheus.yml
Instant query from CLI
promtool query instant http://localhost:9090 'topk(5, sum by(pod)(rate(container_cpu_usage_seconds_total[5m])))'
Find cardinality offenders in TSDB
promtool tsdb analyze /prometheus --limit 20