The problem
Loki alerting rules are metric queries. An alert reports that count_over_time({app="api"} |= "ERROR" [5m]) was above zero, but not which lines matched, so whoever is on call opens Grafana and rebuilds the query by hand. grafana/loki#5844 "Add log message in alert" has 113 reactions and has been open since April 2022. It proposes to infer the log query from the metric query. The Loki product manager acknowledged it in April 2025; the last comment, from March 2026, asks for updates. On Stack Overflow, How to send alert for every error in my logs using Promtail / Loki - AlertManager? has about 15,000 views.
The workaround in the thread is to extract the message into a label with | pattern and group by it. That works for short, low-cardinality messages and gives one line per alert. Grafana-managed alerts on a Loki data source have the same gap.
How it works
loki-alert-lines is a Go program without dependencies outside the standard library. It accepts alerts on /api/v1/alerts and /api/v2/alerts, so the ruler's alertmanager_url can point at it, and forwards them to one or more Alertmanagers.
For each firing alert it reads the rule expression from the generator URL. It finds the log range expression inside the metric query (stream selector, pipeline and range, respecting strings and nested parentheses), drops | unwrap stages, and appends a label filter for every label of the alert instance except alertname and the rule's static labels, which it reads from the ruler API. It runs that log query over the window from startsAt minus the range up to now and writes the newest lines into a log_lines annotation. A template can then show {{ .Annotations.log_lines }} in Slack, mail or any other receiver. A logs_query annotation on the rule replaces the derived query when the derivation does not fit.
For Grafana-managed rules the generator URL contains the rule UID. The proxy reads the rule through Grafana's provisioning API, picks the query on a Loki data source and runs the log query through Grafana's data source proxy, so it needs no direct access to Loki.
Example
From the end-to-end test against Loki 3.7.9 and Alertmanager v0.35.0. The rule sum by (msg) (count_over_time({job="json"} | json | level="error" [1m])) > 2 fired for msg="disk full", and the alert arrived in Alertmanager with:
log_lines_query: {job="json"} | json | level="error" | msg="disk full"
log_lines:
2026-11-13T05:17:58.9541812Z {"level":"error","msg":"disk full","n":1}
2026-11-13T05:17:59.9541812Z {"level":"error","msg":"disk full","n":2}
2026-11-13T05:18:00.9541812Z {"level":"error","msg":"disk full","n":3}The line with msg="other" and the line with level="info" are not in the list.
Testing
Unit tests cover the LogQL extraction (unwrap, offsets, brackets and braces inside strings, ip() filters, Grafana's $__auto range), the label filters, the forwarding with tenant and auth headers, and the Grafana path against a fake Grafana. The end-to-end test starts Loki with auth_enabled: true and the ruler, Alertmanager with a webhook receiver and, in CI, Grafana with a provisioned Loki data source, alert rule and Alertmanager contact point. It pushes log lines, waits for the alerts and checks the annotations in Alertmanager, the webhook notification and the proxy's metrics. CI runs it against Loki 2.9.17, 3.4.6, 3.7.9 and latest, Alertmanager v0.27.0, v0.28.1, v0.35.0 and latest, and Grafana 12.4.15, 13.3.1 and latest.
The real servers showed two things the unit tests had not. Loki 2.9.17 and 3.7.9 do not write the Prometheus style /graph?g0.expr= generator URL but a Grafana Explore link with the expression in a JSON parameter. The first version did not read it and silently fell back to the ruler API; a metric for the source of each query now makes that visible. And Grafana sent firing alerts with endsAt set to the zero time (year 1), which the proxy took for resolved alerts.
Limits
The LogQL handling is not a full parser. With two log queries in one rule, such as a ratio, it uses the first. Labels that only exist after the log query, for example from label_replace, become filters that match nothing, so such rules need logs_query. Alertmanager does not send a new notification only because the lines changed. Not tested: Mimir or Cortex as Alertmanager, highly available rulers against a real cluster, and real Slack or mail receivers.
The code is at github.com/Jwrede/loki-alert-lines.