Skip to content
Grafana Loki & LogQL

Grafana Loki & LogQLSkill

by Grafana Labs
Released
Apache-2.0
Repository Docs
markdown Development

Summary

Grafana's official Loki skill: write LogQL that returns rows instead of timeouts, ship logs through Alloy, and reason about a store that indexes labels rather than log text.

Features

  • Full LogQL reference: stream selectors, line filters, parsers, label filters and formatting stages
  • Metric queries including rate, count_over_time, topk and unwrapped quantile_over_time
  • Grafana Alloy pipelines for file tailing, Kubernetes pod discovery and JSON parse stages
  • Raw /loki/api/v1/push payload for shipping logs without a collector
  • Loki read/write path architecture: distributor, ingester, querier, query frontend, compactor
  • Worked examples for error-rate alerts, slow requests, 5xx breakdowns and credential-leak detection

Install This Skill

Add this skill to your favorite AI agent in a few steps.

Any AI agent

This skill is plain instructions — it works with any assistant that accepts custom instructions or system prompts.

  1. Copy the skill content with the button below.
  2. Paste it into your agent's instruction file or system prompt (for example AGENTS.md, .cursorrules, or a custom instructions field).
  3. Ask the agent to apply the skill whenever the task matches.

Skill Content

Markdown Content

Copy this content and use it with your preferred AI agent

---
name: loki
license: Apache-2.0
description: >
  Grafana Loki log aggregation and LogQL query language. Covers LogQL syntax (log queries, metric queries,
  label matchers, line filters, parsers: json/logfmt/pattern/regexp/unpack, label filters, line_format),
  Loki architecture, log ingestion via Alloy/Promtail/Fluent Bit, structured metadata, and Logs Drilldown.
  Use when writing LogQL queries, configuring Loki, troubleshooting log pipelines, or analyzing logs.
---

# Grafana Loki - Log Aggregation

> **Docs**: https://grafana.com/docs/loki/latest/

Indexes only metadata (labels), not full log content — dramatically cheaper than full-text search systems.

## LogQL Quick Reference

### Log Stream Selector (required in every query)

```logql
{app="nginx"}                        # exact match
{app!="nginx"}                       # not equal
{app=~"nginx|apache"}               # regex match
{app!~"debug.*"}                     # regex not match
{app="nginx", env="prod"}           # AND (multiple labels)
```

### Line Filters (pipeline stage 1 - put first for performance)

```logql
{app="nginx"} |= "error"            # contains string
{app="nginx"} != "info"             # does not contain
{app="nginx"} |~ "error|warn"       # regex match
{app="nginx"} !~ "health.*check"    # regex not match
{app="nginx"} |= `"status":5`       # backtick avoids escaping
```

### Parsers

```logql
# JSON
{app="api"} | json
{app="api"} | json status="http_status", path="request.path"

# Logfmt
{app="api"} | logfmt
{app="api"} | logfmt --strict
{app="api"} | logfmt --keep-empty

# Pattern (positional, _ discards)
{app="nginx"} | pattern `<ip> - - <_> "<method> <uri> <_>" <status> <bytes>`

# Regexp (named capture groups)
{app="nginx"} | regexp `(?P<method>\w+) (?P<path>\S+) HTTP/(?P<version>\S+)`

# Unpack (unwrap Promtail packed labels)
{app="api"} | unpack
```

### Label Filters (after parsers)

```logql
{app="api"} | json | status >= 500
{app="api"} | json | status == 200 and method != "OPTIONS"
{app="api"} | logfmt | duration > 1s
{app="api"} | json | level =~ "error|warn"
{app="api"} | json | bytes > 20MB
{app="api"} | json | path != "/healthz"
```

### Line Format

```logql
{app="api"} | json | line_format "{{.method}} {{.path}} -> {{.status}} ({{.duration}})"
{app="api"} | logfmt | line_format `{{.level | upper}}: {{.msg}}`
```

### Label Format

```logql
{app="api"} | logfmt | label_format new_name=old_name
{app="api"} | logfmt | label_format severity=level, svc=app
{app="api"} | logfmt | label_format msg=`{{.level}}: {{.message}}`
```

### Drop/Keep Labels

```logql
{app="api"} | json | drop filename, level="debug"
{app="api"} | json | keep level, status, method
```

### Decolorize

```logql
{app="cli-tool"} | decolorize
```

## Metric Queries

### Log Range Aggregations

```logql
# Requests per second
rate({app="nginx"}[5m])

# Total log lines in window
count_over_time({app="nginx"}[1h])

# Bytes per second
bytes_rate({app="nginx"}[5m])

# Total bytes
bytes_over_time({app="nginx"}[1h])

# Returns 1 if no logs in range (for absence alerting)
absent_over_time({app="nginx"}[5m])
```

### Aggregation

```logql
# Error rate by service
sum(rate({env="prod"} |= "error" [5m])) by (app)

# Top 5 most active services
topk(5, sum(rate({env="prod"}[5m])) by (app))

# Total errors across all services
sum(count_over_time({env="prod"} |= "error" [5m]))
```

### Unwrapped Range Aggregations (numeric values from logs)

```logql
# Average request duration from logfmt
avg_over_time({app="api"} | logfmt | unwrap duration [5m])

# 95th percentile latency
quantile_over_time(0.95, {app="api"} | logfmt | unwrap duration [5m]) by (app)

# Sum of bytes from JSON logs
sum_over_time({app="api"} | json | unwrap bytes [5m])

# With conversion (duration string → seconds)
avg_over_time({app="api"} | logfmt | unwrap duration | duration_seconds [5m])
```

### Offset Modifier

```logql
# Compare current rate vs 1 hour ago
rate({app="nginx"}[5m]) / rate({app="nginx"}[5m] offset 1h)
```

## Practical Examples

### Error rate alert query
```logql
sum(rate({env="prod"} |= "error" [5m])) by (service)
/
sum(rate({env="prod"}[5m])) by (service)
> 0.05
```

### Slow requests
```logql
{app="api"} | logfmt | duration > 1s | line_format "SLOW: {{.method}} {{.path}} {{.duration}}"
```

### HTTP 5xx errors with details
```logql
{app="nginx"} | pattern `<ip> - - <_> "<method> <uri> <_>" <status> <bytes>` | status >= 500
```

### Credential leak detection
```logql
{namespace="prod"} |~ `https?://\w+:\w+@`
```

## Sending Logs to Loki

### Via Grafana Alloy

```alloy
loki.source.file "app" {
  targets    = [{__path__ = "/var/log/app/*.log", job = "app"}]
  forward_to = [loki.process.parse.receiver]
}

loki.process "parse" {
  forward_to = [loki.write.cloud.receiver]
  stage.json {
    expressions = { level = "level", msg = "message" }
  }
  stage.labels {
    values = { level = "" }
  }
  stage.drop {
    expression = ".*healthcheck.*"
  }
}

loki.write "cloud" {
  endpoint {
    url = "https://logs-xxx.grafana.net/loki/api/v1/push"
    basic_auth {
      username = sys.env("LOKI_USER")
      password = sys.env("GRAFANA_API_KEY")
    }
  }
  external_labels = { cluster = "prod" }
}
```

### Via Kubernetes (Alloy DaemonSet)

```alloy
discovery.kubernetes "pods" {
  role = "pod"
}

loki.source.kubernetes "pods" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [loki.write.cloud.receiver]
}
```

### Loki HTTP Push API

```bash
curl -X POST https://logs-xxx.grafana.net/loki/api/v1/push \
  -u "user:apikey" \
  -H 'Content-Type: application/json' \
  -d '{
    "streams": [{
      "stream": { "app": "myapp", "env": "prod" },
      "values": [
        ["1609459200000000000", "log line here"]
      ]
    }]
  }'
```

## Architecture

```
Push path:  Client → Distributor → Ingester (WAL) → Object Storage (chunks)
Read path:  Query → Query Frontend → Querier → Ingester + Store (chunks)
```

**Components:**
- **Distributor**: Validates and hashes incoming log streams
- **Ingester**: Buffers chunks in memory, flushes to object storage
- **Querier**: Executes LogQL queries
- **Query Frontend**: Caches, splits, and parallelizes queries
- **Compactor**: Manages retention and deduplication

## References

- [LogQL Reference](references/logql.md)
- [Configuration](references/configuration.md)
- [Sending Data](references/send-data.md)

Description

Loki is Grafana's log database, and its whole cost argument rests on one design decision: it indexes labels, not the contents of your log lines. That decision is also why a LogQL query written like an Elasticsearch query gets slow — the label selector does the narrowing, and everything after the pipe is a scan.

This is the skill Grafana Labs maintains for that gap. It gives a coding agent a working LogQL reference organised in the order a query is actually built: the stream selector that every query requires, line filters (|=, !=, |~, !~) placed first because they cut the scan, the five parsers (json, logfmt, pattern, regexp, unpack), label filters that compare typed values such as status >= 500 or duration > 1s, and the line_format / label_format / drop / keep stages that reshape what comes back.

The metric half is covered to the same depth: rate, count_over_time, bytes_rate and absent_over_time for log-range aggregations, topk and sum ... by for grouping, and the unwrapped aggregations (quantile_over_time, avg_over_time, sum_over_time) that pull numbers out of log payloads — including the duration_seconds conversion that trips people up. Worked examples cover an error-rate alert expression, slow-request extraction, HTTP 5xx breakdown via pattern, and a regex that finds credentials leaked into URLs.

Ingestion is the other half. The skill carries ready Grafana Alloy configurations for file tailing with a JSON parse stage, label promotion and a healthcheck stage.drop, a Kubernetes DaemonSet variant using discovery.kubernetes, and the raw /loki/api/v1/push HTTP payload for anything that has to speak to Loki directly. A short architecture section names the distributor, ingester, querier, query frontend and compactor and what each one is responsible for on the read and write paths.

Apache-2.0, part of the grafana/skills repository, and installable with npx skills add grafana/skills or as a Claude Code plugin marketplace. Useful for anyone writing LogQL by hand, debugging a log pipeline that shows nothing in Explore, or trying to work out which label blew up their stream count.

Install

Related Skills

Skill: Grafana Alloy Collector Configs

by Grafana Labs

Grafana's official Alloy skill: write one collector config that carries metrics, logs, traces and profiles, validate it locally, and prove samples are actually leaving the box.

Development

Skill: Grafana Tempo & TraceQL

by Grafana Labs

Grafana's official Tempo skill: stand up a tracing backend that needs nothing but object storage, then write TraceQL that finds the slow span instead of scrolling for it.

Development

Skill: Datadog Logs

by Datadog

Search Datadog logs from an agent and keep the bill under control — query syntax, exclusion filters, log-based metrics, archives and PII scrubbing.

Development

Skill: PromQL Query Patterns

by Grafana Labs

Grafana's official PromQL skill: write, validate, and optimise Prometheus queries, and hunt down the label that blew up your cardinality.

Development
Browse all skills →