Skip to content

Commit b22fc6f

Browse files
authored
feat: [#29] add CRE with detection of long line ingestion error in loki (#36)
* feat: [#29] add CRE with detection of long line ingestion error in loki * chore: [#29] emphasize alloy log is the source, set author of the rule
1 parent c8ab120 commit b22fc6f

File tree

1 file changed

+58
-0
lines changed

1 file changed

+58
-0
lines changed
Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
rules:
2+
- metadata:
3+
id: URE4JfAmWXG8qee2kyZcps
4+
gen: 1
5+
kind: "prequel"
6+
version: "1.0.0"
7+
cre:
8+
id: CRE-2025-0070
9+
title: Loki Log Line Exceeds Max Size Limit
10+
severity: 3
11+
category: log-processing-problems
12+
author: dosmanak
13+
description: |
14+
Alloy detects the Loki is dropping log lines because they exceed the configured maximum line size.
15+
This typically indicates that applications are emitting extremely long log entries, which Loki is configured to reject by default.
16+
cause: |
17+
Loki has a `max_line_size` limit (defaulting to 256KB) to prevent excessively large log entries from consuming too many resources.
18+
When a log line from an application exceeds this limit, Loki rejects it, leading to data loss for that specific log entry.
19+
impact: |
20+
Critical log data or important diagnostic information contained within the excessively long lines is lost, making debugging
21+
and monitoring challenging. This can lead to blind spots in observability for systems generating large log entries.
22+
mitigation: |
23+
To resolve this issue, you can adjust Loki's runtime configuration to either:
24+
25+
1. **Increase `max_line_size`**: Allow Loki to accept larger log lines. Be cautious when increasing this significantly,
26+
as it can impact Loki's performance and memory consumption.
27+
2. **Enable `max_line_size_truncate`**: Configure Loki to automatically truncate log lines that exceed the limit
28+
instead of dropping them entirely. This preserves the beginning of the log message.
29+
30+
You can apply these changes via Loki's runtime configuration overrides, typically in a YAML file like this:
31+
32+
```
33+
---
34+
overrides:
35+
limits_config:
36+
max_line_size: 10485760 # Set to 10MB (10 * 1024 * 1024 bytes)
37+
max_line_size_truncate: true
38+
```
39+
40+
Ensure that your Loki deployment is configured to load this runtime override file.
41+
tags:
42+
- alloy
43+
- loki
44+
- logs
45+
- line-too-long
46+
- observability
47+
- grafana
48+
references:
49+
- "https://grafana.com/docs/grafana-cloud/send-data/logs/troubleshoot/#line-too-long"
50+
rule:
51+
set: # Using 'set' for single event matching, as it's a single log line detection
52+
event:
53+
source: Alloy log
54+
match:
55+
- regex: >
56+
level=error msg="final error sending batch" component_path=/ component_id=loki\.write\.endpoint component=client host=.*?
57+
status=400 tenant=.*? error="server returned HTTP status 400 Bad Request \(400\): Max entry size \'(\d+)\' bytes
58+
exceeded for stream \'\{.*?\}\' while adding an entry with length \'(\d+)\' bytes"

0 commit comments

Comments
 (0)