PromQL Deep Dive - Rates, Aggregation, and Vector Matching

by Bala

How rate/irate/increase actually work on counters, aggregation operators, vector matching, subqueries, and the pitfalls that produce wrong dashboards

Introduction

Most PromQL mistakes aren’t syntax errors - they’re queries that run fine and produce a plausible-looking, wrong number. Understanding what rate() actually does with counter resets, how vector matching decides which series to pair up, and when a subquery is genuinely needed instead of a hack are the difference between a dashboard you can trust and one that quietly lies during an incident.

Instant Vectors vs Range Vectors

# Instant vector: one sample per series, "right now"
http_requests_total

# Range vector: a window of samples per series over time
http_requests_total[5m]

A range vector selector ([5m]) can’t be graphed or alerted on directly - it’s raw input to a function like rate() that reduces it back down to an instant vector. This is the single most common beginner error: writing http_requests_total[5m] in a Grafana panel and getting no output, because a range vector isn’t a valid top-level query result.

rate(), irate(), and increase()

# Per-second average rate of increase over the window
rate(http_requests_total[5m])

# Total increase over the window (rate * window duration)
increase(http_requests_total[5m])

# Instantaneous rate using only the last two data points in the window
irate(http_requests_total[5m])

All three are counter-only functions - http_requests_total only ever goes up (or resets to 0 on a process restart), and these functions automatically detect and correctly handle counter resets: if the current sample is lower than the previous one, Prometheus treats that as a reset and adjusts the calculation instead of producing a negative rate. Using rate()/irate()/increase() on a gauge (a value that legitimately goes up and down, like current memory usage) produces meaningless numbers, since there’s no “reset” to detect - gauges use delta()/deriv() instead.

rate() averages over the whole window and is what you almost always want for alerting and dashboards - smooth, resistant to a single noisy scrape. irate() uses only the last two points and reacts instantly to spikes, which sounds better but makes it useless for alerting (a single slow scrape interval creates a spike that immediately vanishes) - reserve irate() for ad-hoc, high-resolution graphing, never for an alert’s condition.

rate() needs at least two samples inside the window to compute anything - a window shorter than roughly 4x the scrape interval risks gaps or NaN for series that happen to align unluckily with scrape timing; a common rule of thumb is a range at least 4x the scrape interval ([5m] for a 15s-30s scrape interval, for instance, not [1m]).

Aggregation Operators

# Total request rate across all instances, grouped by service
sum(rate(http_requests_total[5m])) by (service)

# 95th percentile latency per service, from a histogram
histogram_quantile(0.95,
  sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)

# Top 5 pods by memory usage
topk(5, container_memory_working_set_bytes)

# Number of distinct instances reporting per job
count(up == 1) by (job)

by (service) keeps only the listed labels after aggregating, dropping everything else (like instance or pod) - the opposite, without (instance), keeps every label except the ones listed. Picking the wrong one is a frequent source of “why did my labels disappear” confusion: use by when you know exactly which labels should survive; use without when you want to keep everything except a couple of high-cardinality ones.

histogram_quantile() requires le (the bucket boundary label Prometheus histograms expose) to survive the by clause - dropping it produces either an error or a meaningless result, since the function needs the full set of bucket boundaries per series to interpolate a quantile.

Vector Matching

# Error rate as a percentage: errors / total
sum(rate(http_requests_total{code=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)

# Many-to-one: attach a single "team" label from a mapping series
# onto every per-instance series
rate(http_requests_total[5m]) * on(service) group_left(team) service_team_mapping

A binary operator between two vectors matches series by their shared labels by default - if the two sides don’t have identical label sets (a common case: one side aggregated by (service), the other still has instance too), the match silently fails to find any series and the query returns empty, with no error. on(service) narrows matching to just the listed label(s) explicitly; ignoring(instance) does the inverse (match on everything except the listed labels). group_left/group_right are required whenever the match is many-to-one/one-to-many rather than one-to-one (attaching one metadata series’s labels onto many per-instance series, as above) - without it, Prometheus refuses the query with a “many-to-one matching” error rather than silently picking one arbitrarily.

Subqueries

# Max rate of requests over the last hour, sampled every minute
max_over_time(rate(http_requests_total[5m])[1h:1m])

A subquery ([1h:1m]) re-evaluates the inner expression repeatedly over a range and resolution, turning an instant-vector expression back into a range vector another function can operate on - the case for it is specifically “I need rate() computed at each point over a long window, then aggregated across that window,” which a plain range-vector selector can’t express since rate() itself already needs a range vector as input. Subqueries are more expensive to evaluate than an equivalent recording rule precomputing the same thing on a schedule - reach for a recording rule instead of a subquery for anything queried repeatedly (a dashboard panel refreshed constantly), and keep subqueries for genuinely ad-hoc exploration.

Common Pitfalls

  1. Aggregating before rate, not after - rate(sum(http_requests_total)[5m]) is invalid and, if you work around the error by summing first with no rate, you lose the automatic counter-reset handling rate() provides per-series. Always rate() each series individually, then sum() the results: sum(rate(http_requests_total[5m])).
  2. Using irate() in an alert rule - it reacts to two-point noise, producing alerts that fire and clear within a single scrape interval. Use rate() for anything an AlertmanagerConfig or PrometheusRule evaluates.
  3. Forgetting that up == 0 means “not scraped,” not “problem” - a target that’s up == 0 because it was intentionally scaled to zero, or hasn’t started yet, looks identical in PromQL to a genuine outage; combine with other signals before alerting purely on up.
  4. High-cardinality by() clauses - grouping by a label with thousands of distinct values (a raw pod name in a cluster with heavy autoscaling churn, or worse, a user_id) creates a time series per combination, which is a real, sometimes severe, cost on Prometheus’s memory and query performance.

Conclusion

The functions that look interchangeable at a glance - rate() vs irate(), by vs without, a subquery vs a recording rule - each have a real, different correctness or cost tradeoff, and picking the wrong one tends to fail silently rather than with an error. Reading the actual semantics once is cheaper than debugging a wrong dashboard during an incident.

Additional Resources