prometheus commands

30 commands, each with a one-line explanation. Click a command to copy it, or its title for details.

Commands 30 shown

  • Query Prometheus via curl
    curl -sG 'http://prometheus:9090/api/v1/query' --data-urlencode 'query=up' | jq .

    Ad-hoc PromQL query from CLI

    monitoring

  • Instant query via promtool
    promtool query instant http://prometheus:9090 'up{job="kubelet"}'

    Run an instant PromQL query using promtool

    monitoring · debug

  • Range query via curl
    curl -sG 'http://prometheus:9090/api/v1/query_range' --data-urlencode 'query=rate(http_requests_total[5m])' --data-urlencode 'start=2024-01-01T00:00:00Z' --data-urlencode 'end=2024-01-01T01:00:00Z' --data-urlencode 'step=60s' | jq .

    Query Prometheus for a time range via HTTP API

    monitoring

  • Check alert rules
    promtool check rules rules.yml

    Validate Prometheus alert/recording rule files for syntax

    monitoring

  • Silence active alert
    amtool silence add --alertmanager.url=http://alertmanager:9093 alertname=<name> --duration=2h --comment='Maintenance'

    Create a silence to mute alert notifications during maintenance

    alertmanager · monitoring

  • List silences
    amtool silence query --alertmanager.url=http://alertmanager:9093

    Show all active silences in Alertmanager

    alertmanager · monitoring

  • Expire silence
    amtool silence expire <id> --alertmanager.url=http://alertmanager:9093

    Immediately end an active silence

    alertmanager · monitoring

  • Check alertmanager config
    amtool check-config alertmanager.yml

    Validate Alertmanager configuration file

    alertmanager · monitoring

  • List active alerts
    amtool alert query --alertmanager.url=http://alertmanager:9093

    Show all firing alerts in Alertmanager

    alertmanager · monitoring

  • Reload Prometheus config
    curl -X POST http://prometheus:9090/-/reload

    Hot-reload Prometheus configuration without restart

    monitoring

  • Query instant vector
    curl -sG 'http://prometheus:9090/api/v1/query' --data-urlencode 'query=up' | jq '.data.result'

    Run an instant PromQL query via HTTP API

    monitoring

  • Range query for graph data
    curl -sG 'http://prometheus:9090/api/v1/query_range' --data-urlencode 'query=rate(http_requests_total[5m])' --data-urlencode 'start=1h' --data-urlencode 'end=now' --data-urlencode 'step=60'

    Fetch time-series data over a range for a rate metric

    monitoring

  • Top N series by value
    topk(10, sum by (pod) (rate(container_cpu_usage_seconds_total[5m])))

    PromQL: top 10 pods by CPU usage rate

    monitoring

  • Disk usage alert rule
    100 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100) > 85

    PromQL expression: fire when root disk > 85% used

    monitoring

  • Reload Prometheus config
    curl -X POST http://prometheus:9090/-/reload

    Hot reload Prometheus config and rules without restart

  • List active alerts
    curl -s http://prometheus:9090/api/v1/alerts | jq '.data.alerts[] | select(.state=="firing") | {name: .labels.alertname, severity: .labels.severity}'

    Show only firing alerts with name and severity

    monitoring

  • List all metric names
    curl -s http://prometheus:9090/api/v1/label/__name__/values | jq '.data | length, .[0:20]'

    Count total metric names and preview first 20

    debug

  • Series cardinality per job
    count by (job) ({__name__=~".+"})

    PromQL: count active time series grouped by job label

    monitoring

  • Silence an alert
    amtool silence add alertname=HighMemory --duration=2h --comment='Investigating'

    Create an Alertmanager silence for 2 hours

    monitoring

  • Check TSDB status
    curl -s http://prometheus:9090/api/v1/status/tsdb | jq '{head_chunks: .data.headStats.numChunks, series: .data.headStats.numSeries}'

    View TSDB head stats: chunks and active series

    monitoring

  • 95th percentile latency from histogram
    histogram_quantile(0.95, sum by(le,job)(rate(http_request_duration_seconds_bucket[5m])))

    p95 latency per job; note: always aggregate _bucket by le before histogram_quantile

    performance · observability · monitoring

  • Predict disk full window
    predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4*3600) < 0

    alert when linear trend predicts disk full within 4h based on last 6h

    disk · monitoring · troubleshooting

  • Detect dead target with absent
    absent(up{job="node"} == 1)

    fires when no series match; catches a job that vanished from discovery entirely

    monitoring · troubleshooting · observability

  • Rewrite label with label_replace
    label_replace(node_uname_info, "host", "$1", "nodename", "(.+)")

    extract/derive a new label via regex capture; useful for joins and tidy dashboards

    observability · monitoring

  • Max of per-second rate over window
    max_over_time(rate(node_network_receive_bytes_total[5m])[1h:1m])

    subquery: peak 5m rate sampled each minute over the last hour; spot bursts

    performance · network · observability

  • Aggregate keeping all labels but instance
    sum without(instance) (rate(http_requests_total{code=~"5.."}[5m]))

    without() keeps every label except instance; cleaner than listing many by() labels

    observability · monitoring

  • Recording rule for expensive ratio
    groups: - name: slo rules: - record: job:request_errors:ratio5m expr: sum by(job)(rate(http_requests_total{code=~"5.."}[5m])) / sum by(job)(rate(http_requests_total[5m]))

    precompute SLO error ratio to keep dashboards and alerts fast

    gitops · observability · performance

  • Validate rules and config with promtool
    promtool check rules rules.yml && promtool check config prometheus.yml

    lint before reload; run in CI to catch broken expr/yaml before deploy

    ci · gitops · troubleshooting

  • Instant query from CLI
    promtool query instant http://localhost:9090 'topk(5, sum by(pod)(rate(container_cpu_usage_seconds_total[5m])))'

    run PromQL without a browser; great for scripted top-N investigations

    debug · troubleshooting · performance

  • Find cardinality offenders in TSDB
    promtool tsdb analyze /prometheus --limit 20

    lists highest-cardinality metrics and label pairs; first stop for memory blowups

    performance · troubleshooting · observability