Get a latency quantile from logs

p99 latency per route from a numeric log field. Note: unwrap needs a numeric label or field

quantile_over_time(0.99, {app="api"} | json | unwrap latency_ms [5m]) by (route)

tool
loki7 commands
tags
lokilogqlperformanceobservability

More loki commands

  • sum(count_over_time({namespace="prod"} |~ "timeout|deadline exceeded" [1m]))

    Count error lines that match a regex, per minute, across a namespace. |~ means regex match

  • logcli query '{job="app/api"} |= "panic"' --since=1h --limit=200 --output=jsonl

    Query the last hour from the CLI with a line limit and JSONL output. Set LOKI_ADDR first

  • curl -sG "$LOKI_ADDR/loki/api/v1/query_range" --data-urlencode 'query=sum(rate({app="api"}[5m]))' --data-urlencode 'step=60' | jq '.data.result'

    Run a range metric query over HTTP and get the result with jq. step is in seconds

  • curl -s -H 'Content-Type: application/json' -XPOST "$LOKI_ADDR/loki/api/v1/push" -d '{"streams":[{"stream":{"app":"test"},"values":[["'$(date +%s%N)'","hello loki"]]}]}'

    Push one log entry by hand to check ingestion. Note: the timestamp must be in nanoseconds since epoch

  • {app="api"} |= "error" | json | line_format "{{.level}} {{.msg}}"

    Filter by substring, parse JSON and print only the fields you need. Note: | json goes before line_format

all 7 loki commands