|
| 1 | +rules: |
| 2 | + - metadata: |
| 3 | + id: URE4JfAmWXG8qee2kyZcps |
| 4 | + gen: 1 |
| 5 | + kind: "prequel" |
| 6 | + version: "1.0.0" |
| 7 | + cre: |
| 8 | + id: CRE-2025-0070 |
| 9 | + title: Loki Log Line Exceeds Max Size Limit |
| 10 | + severity: 3 |
| 11 | + category: log-processing-problems |
| 12 | + author: dosmanak |
| 13 | + description: | |
| 14 | + Alloy detects the Loki is dropping log lines because they exceed the configured maximum line size. |
| 15 | + This typically indicates that applications are emitting extremely long log entries, which Loki is configured to reject by default. |
| 16 | + cause: | |
| 17 | + Loki has a `max_line_size` limit (defaulting to 256KB) to prevent excessively large log entries from consuming too many resources. |
| 18 | + When a log line from an application exceeds this limit, Loki rejects it, leading to data loss for that specific log entry. |
| 19 | + impact: | |
| 20 | + Critical log data or important diagnostic information contained within the excessively long lines is lost, making debugging |
| 21 | + and monitoring challenging. This can lead to blind spots in observability for systems generating large log entries. |
| 22 | + mitigation: | |
| 23 | + To resolve this issue, you can adjust Loki's runtime configuration to either: |
| 24 | +
|
| 25 | + 1. **Increase `max_line_size`**: Allow Loki to accept larger log lines. Be cautious when increasing this significantly, |
| 26 | + as it can impact Loki's performance and memory consumption. |
| 27 | + 2. **Enable `max_line_size_truncate`**: Configure Loki to automatically truncate log lines that exceed the limit |
| 28 | + instead of dropping them entirely. This preserves the beginning of the log message. |
| 29 | +
|
| 30 | + You can apply these changes via Loki's runtime configuration overrides, typically in a YAML file like this: |
| 31 | +
|
| 32 | + ``` |
| 33 | + --- |
| 34 | + overrides: |
| 35 | + limits_config: |
| 36 | + max_line_size: 10485760 # Set to 10MB (10 * 1024 * 1024 bytes) |
| 37 | + max_line_size_truncate: true |
| 38 | + ``` |
| 39 | +
|
| 40 | + Ensure that your Loki deployment is configured to load this runtime override file. |
| 41 | + tags: |
| 42 | + - alloy |
| 43 | + - loki |
| 44 | + - logs |
| 45 | + - line-too-long |
| 46 | + - observability |
| 47 | + - grafana |
| 48 | + references: |
| 49 | + - "https://grafana.com/docs/grafana-cloud/send-data/logs/troubleshoot/#line-too-long" |
| 50 | + rule: |
| 51 | + set: # Using 'set' for single event matching, as it's a single log line detection |
| 52 | + event: |
| 53 | + source: Alloy log |
| 54 | + match: |
| 55 | + - regex: > |
| 56 | + level=error msg="final error sending batch" component_path=/ component_id=loki\.write\.endpoint component=client host=.*? |
| 57 | + status=400 tenant=.*? error="server returned HTTP status 400 Bad Request \(400\): Max entry size \'(\d+)\' bytes |
| 58 | + exceeded for stream \'\{.*?\}\' while adding an entry with length \'(\d+)\' bytes" |
0 commit comments