🌊 Dissect suggestions (#242377)

## Add Dissect Pattern Suggestion Support to Streams Processing

### Summary
This PR adds automatic dissect pattern generation capabilities to the
Streams processing pipeline, complementing the existing grok pattern
suggestions. Dissect patterns provide faster log parsing for structured
logs with simple delimiters (vs regex-based grok).

### What was added

#### New Package: `@kbn/dissect-heuristics`
- **Core algorithm** (`extractDissectPatternDangerouslySlow`): Analyzes
sample log messages to automatically extract dissect patterns
- 6-step pipeline: whitespace normalization → delimiter detection →
delimiter tree building → field extraction → modifier detection →
pattern generation
- Supports dissect modifiers: right padding (`->`), named skip (`?`),
empty skip (`{}`)

- **LLM Review Integration**: Maps generic field names to ECS-compliant
field names
  - `getReviewFields`: Prepares field metadata for LLM review
- `getDissectProcessorWithReview`: Applies LLM suggestions to rename
fields and handle multi-column field grouping
  - `ReviewDissectFieldsPrompt`: Structured prompt for LLM field mapping

- **Message Grouping**: Re-exports `groupMessagesByPattern` from
`@kbn/grok-heuristics` for consistent message clustering

#### Server-Side API
- **New endpoint**: `POST
/internal/streams/{name}/processing/_suggestions/dissect`
  - Input: connector ID, sample messages, review fields
  - Output: SSE stream with dissect processor configuration
- Handler (dissect_suggestions_handler.ts): Orchestrates LLM review and
field mapping with OTEL/ECS field name resolution

#### Client-Side Integration
- **React hook** (`useDissectPatternSuggestion`): 
  - Groups messages by pattern using `groupMessagesByPattern`
  - Extracts dissect pattern from the largest message group
  - Calls LLM for field review
  - Simulates processor to validate results
  - Includes telemetry tracking for AI suggestion latency

### Architecture
Follows the same pattern as existing grok suggestions:
1. Client groups similar log messages
2. Heuristic algorithm extracts pattern from largest group
3. LLM reviews and maps fields to ECS/OTEL standards (can decide to
group fields, turn fields into static parts of the pattern, can decide
to skip fields)
4. Simulation validates the processor before applying

### Open questions / considerations

* I forked a bunch of stuff from the grok implementation, theoretically
some redundancy could be avoided, but I'm not sure how much it would
help. For both client and server I abstracted out some base helpers, but
I didn't go so far to invent a whole new subsystem for pattern
suggestions. Maybe it's worth it, not sure.
* I'm using the same pre-grouping used for grok, then just go with the
biggest group, since if there are completely different message patterns,
you are out of luck anyway with dissect. We could try to make the base
logic smarter, but not sure how
* When parsing date patterns, it's very common that they are captured
with multiple groups, like `%{+timestamp}-%{+timestamp}-%{+timestamp}`.
This works fine, but it means that with the default `' '` append
separator, the resulting custom timestamp column becomes a non-standard
date format, which is not captured by the date format suggestion logic
we have in place. Maybe we can make that smarter, that would be great
anyway
* Added new tracking events for dissect patterns, could also be a param
on the existing one, but I wanted to stay backwards compatible
* The dissect processor could need some love, e.g. a better editor
experience, syntax highlighting, automatic multi-line preview, maybe
even highlighting like grok... But I think it is out of scope for this
PR
* Sometimes the AI messes up and puts static values in places where they
don't belong, breaking matches. We might be able to improve on that, but
it doesn't happen a ton, so I didn't go too far on this. I could imagine
a simulation feedback loop where we try to use the generated pattern, if
it doesn't have matches give it back to the LLM and let it try again

<details>

<summary>Click to expand eval for loghub data</summary>

```
Getting suggestions...

- logs.apache-web: [%{field_1} %{field_2} %{field_3} %{field_4} %{field_5}] [%{field_6}] %{field_7->} %{field_8->} %{field_9}
- logs.hadoop-logs: %{field_1}-%{field_2}-%{field_3} %{field_4},%{field_5} %{field_6} [%{field_7}] %{field_8}: %{field_9} %{field_10} %{field_11} %{field_12} %{field_13}_%{field_14}_%{field_15}_%{field_16}
- logs.bgl-logs: - %{field_1} %{field_2} %{field_3}-%{field_4}-%{field_5}-%{field_6}-%{field_7} %{field_8}-%{field_9}-%{field_10}-%{field_11} %{field_12}-%{field_13}-%{field_14}-%{field_15}-%{field_16} %{field_17} %{field_18} %{field_19} %{field_20} %{field_21} %{field_22} %{field_23} %{field_24}
- logs.health-app-logs: %{field_1}-%{field_2}|%{field_3}_%{field_4}|%{field_5}|%{field_6}
- logs.windows: %{field_1}-%{field_2}-%{field_3} %{field_4}, %{field_5->} %{field_6->} %{field_7->} %{field_8->} %{field_9}
- logs.android: %{field_1}-%{field_2} %{field_3->} %{field_4->} %{field_5->} %{field_6->} %{field_7}: %{field_8}
- logs.thunderbird-logs: - %{field_1} %{field_2} %{field_3->} %{field_4->} %{field_5->} %{field_6->} %{field_7} %{field_8->}(%{field_9->})%{field_10->}[%{field_11->}]: %{field_12->} %{field_13->} %{field_14->} %{field_15}
- logs.proxifier-logs: [%{field_1} %{field_2}] %{field_3} - %{field_4} %{field_5->} %{field_6->} %{field_7->} %{field_8} %{field_9}
- logs.linux: %{field_1} %{field_2} %{field_3} %{field_4} %{field_5}(%{field_6}_%{field_7})[%{field_8}]: %{field_9->} %{field_10}; %{field_11->} %{field_12}
- logs.apache-web: [%{+attributes.custom.timestamp} %{+attributes.custom.timestamp} %{+attributes.custom.timestamp} %{+attributes.custom.timestamp} %{+attributes.custom.timestamp}] [%{severity_text}] %{body.text}
- logs.android: %{+attributes.custom.timestamp}-%{+attributes.custom.timestamp} %{+attributes.custom.timestamp->} %{resource.attributes.process.pid->} %{attributes.process.thread.id->} %{severity_text->} %{attributes.log.logger}: %{body.text}
- logs.windows: %{+attributes.custom.timestamp}-%{+attributes.custom.timestamp}-%{+attributes.custom.timestamp} %{+attributes.custom.timestamp}, %{severity_text->} %{resource.attributes.service.name->} %{body.text}
- logs.health-app-logs: %{+attributes.custom.timestamp}-%{+attributes.custom.timestamp}|Step_%{attributes.log.logger}|%{resource.attributes.process.pid}|%{body.text}
- logs.proxifier-logs: [%{+attributes.custom.timestamp} %{+attributes.custom.timestamp}] chrome.exe - %{attributes.url.domain} %{attributes.event.type->} %{attributes.custom.details}
- logs.thunderbird-logs: - %{attributes.custom.timestamp} %{+attributes.custom.timestamp_text} %{resource.attributes.host.name->} %{+attributes.custom.timestamp_text->} %{+attributes.custom.timestamp_text->} %{+attributes.custom.timestamp_text->} %{attributes.host.hostname} %{attributes.process.name->}(%{attributes.user.name->})%{field_10->}[%{resource.attributes.process.pid->}]: %{field_12->} %{body.text}
- logs.linux: %{+attributes.custom.timestamp} %{+attributes.custom.timestamp} %{+attributes.custom.timestamp} %{attributes.host.hostname} sshd(pam_unix)[%{resource.attributes.process.pid}]: %{+attributes.event.action->} %{+attributes.event.action}; %{body.text}
- logs.bgl-logs: - %{field_1} %{attributes.custom.date} %{+resource.attributes.host.name}-%{+resource.attributes.host.name}-%{+resource.attributes.host.name}-%{+resource.attributes.host.name}-%{+resource.attributes.host.name} %{+attributes.custom.timestamp}-%{+attributes.custom.timestamp}-%{+attributes.custom.timestamp}-%{+attributes.custom.timestamp} %{+attributes.custom.target_host}-%{+attributes.custom.target_host}-%{+attributes.custom.target_host}-%{+attributes.custom.target_host}-%{+attributes.custom.target_host} RAS KERNEL INFO %{body.text}
- logs.hadoop-logs: %{+attributes.custom.timestamp}-%{+attributes.custom.timestamp}-%{+attributes.custom.timestamp} %{+attributes.custom.timestamp},%{+attributes.custom.timestamp} INFO [%{attributes.process.thread.name}] %{attributes.log.logger}: %{attributes.custom.action} %{attributes.custom.component} for application appattempt_%{+attributes.custom.attempt_id}_%{+attributes.custom.attempt_id}_%{+attributes.custom.attempt_id}

Simulate processing...

- logs.apache-web: 1
  → body.text: 2 unique values (e.g., "mod_jk child workerEnv in error state 6", "workerEnv.init() ok /etc/httpd/conf/workers2.properties")
  → severity_text: 2 unique values (e.g., "error", "notice")
  → attributes.custom.timestamp: 38 unique values (e.g., "Fri Nov 14 15:27:00 2025", "Fri Nov 14 15:26:58 2025", "Fri Nov 14 15:26:56 2025", "Fri Nov 14 15:26:53 2025", "Fri Nov 14 15:26:52 2025", "Fri Nov 14 15:26:50 2025", "Fri Nov 14 15:26:49 2025", "Fri Nov 14 15:26:48 2025", "Fri Nov 14 15:26:47 2025", "Fri Nov 14 15:26:45 2025")
- logs.hadoop-logs: 1
  → attributes.process.thread.name: 1 unique values (e.g., "main")
  → attributes.custom.action: 1 unique values (e.g., "Created")
  → attributes.custom.attempt_id: 1 unique values (e.g., "1445144423722 0020 000001")
  → attributes.custom.timestamp: 65 unique values (e.g., "2025 11 14 15:27:01 370", "2025 11 14 15:27:00 070", "2025 11 14 15:26:58 770", "2025 11 14 15:26:57 470", "2025 11 14 15:26:56 170", "2025 11 14 15:26:54 870", "2025 11 14 15:26:53 570", "2025 11 14 15:26:52 270", "2025 11 14 15:26:50 970", "2025 11 14 15:26:49 670")
  → attributes.custom.component: 1 unique values (e.g., "MRAppMaster")
  → attributes.log.logger: 1 unique values (e.g., "org.apache.hadoop.mapreduce.v2.app.MRAppMaster")
- logs.bgl-logs: 1
  → body.text: 1 unique values (e.g., "instruction cache parity error corrected")
  → field_1: 2 unique values (e.g., "1117838573", "1117838570")
  → attributes.custom.date: 1 unique values (e.g., "2005.06.03")
  → attributes.custom.timestamp: 50 unique values (e.g., "2025 11 14 15.27.01.370000", "2025 11 14 15.27.00.070000", "2025 11 14 15.26.58.770000", "2025 11 14 15.26.57.470000", "2025 11 14 15.26.56.170000", "2025 11 14 15.26.54.870000", "2025 11 14 15.26.53.570000", "2025 11 14 15.26.52.270000", "2025 11 14 15.26.50.970000", "2025 11 14 15.26.49.670000")
  → resource.attributes.host.name: 1 unique values (e.g., "R02 M1 N0 C:J12 U11")
  → attributes.custom.target_host: 1 unique values (e.g., "R02 M1 N0 C:J12 U11")
- logs.linux: 0.6818181818181818
  → body.text: 2 unique values (e.g., "user unknown", "logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4")
  → attributes.host.hostname: 1 unique values (e.g., "combo")
  → attributes.event.action: 2 unique values (e.g., "check pass", "authentication failure")
  → resource.attributes.process.pid: 2 unique values (e.g., "19937", "19939")
  → attributes.custom.timestamp: 34 unique values (e.g., "Nov 14 15:27:01", "Nov 14 15:27:00", "Nov 14 15:26:58", "Nov 14 15:26:57", "Nov 14 15:26:56", "Nov 14 15:26:54", "Nov 14 15:26:53", "Nov 14 15:26:52", "Nov 14 15:26:50", "Nov 14 15:26:49")
- logs.android: 1
  → body.text: 22 unique values (e.g., "$printFreezingDisplayLogsopening app wtoken = AppWindowToken{9f4ef63 token=Token{a64f992 ActivityReco...", "HBM brightnessOut =38", "Animating brightness: target=38, rate=200", "HBM brightnessIn =38", "cleanUpApplicationRecordLocked, pid: 5769, restart: false", "cleanUpApplicationRecordLocked, pid: 23484, restart: false", "cleanUpApplicationRecord -- 23484", "cleanUpApplicationRecordLocked, reset pid: 5784, euid: 0", "cleanUpApplicationRecordLocked, pid: 5784, restart: false", "cleanUpApplicationRecord -- 5784")
  → severity_text: 4 unique values (e.g., "D", "I", "V", "W")
  → resource.attributes.process.pid: 4 unique values (e.g., "1702", "23650", "2227", "28601")
  → attributes.custom.timestamp: 95 unique values (e.g., "11 14 15:26:58.770", "11 14 15:26:57.470", "11 14 15:26:52.270", "11 14 15:26:50.970", "11 14 15:26:48.370", "11 14 15:26:45.770", "11 14 15:26:44.370", "11 14 15:26:42.970", "11 14 15:26:41.470", "11 14 15:26:38.870")
  → attributes.process.thread.id: 17 unique values (e.g., "2395", "1820", "1737", "1736", "3693", "17632", "17621", "23689", "2250", "14640")
  → attributes.log.logger: 7 unique values (e.g., "WindowManager", "DisplayPowerController", "ActivityManager", "DisplayManagerService", "AudioManager", "PhoneStatusBar", "PowerManagerService")
- logs.health-app-logs: 1
  → body.text: 10 unique values (e.g., "onExtend:1514038530000 14 0 4", "flush sensor data", "setTodayTotalDetailSteps=1514038440000##7007##548365##8661##12361##27173954", "calculateCaloriesWithCache totalCalories=126775", "processHandleBroadcastAction action:android.intent.action.SCREEN_ON", " getTodayTotalDetailSteps = 1514038440000##6993##548365##8661##12266##27164404", "onStandStepChanged 3579", "onReceive action: android.intent.action.SCREEN_ON", "calculateAltitudeWithCache totalAltitude=240", "REPORT : 7007 5002 150089 240")
  → resource.attributes.process.pid: 1 unique values (e.g., "30002312")
  → attributes.custom.timestamp: 10 unique values (e.g., "20251114 15:27:01:370", "20251114 15:27:00:070", "20251114 15:26:58:770", "20251114 15:26:57:470", "20251114 15:26:56:170", "20251114 15:26:54:870", "20251114 15:26:53:570", "20251114 15:26:52:270", "20251114 15:26:50:970", "20251114 15:26:49:670")
  → attributes.log.logger: 5 unique values (e.g., "LSC", "StandStepCounter", "SPUtils", "ExtSDM", "StandReportReceiver")
- logs.windows: 1
  → body.text: 7 unique values (e.g., "$Loaded Servicing Stack v6.1.7601.23505 with Core: C:\Windows\winsxs\amd64_microsoft-windows-servicin...", "Ending TrustedInstaller finalization.", "Reboot mark refs: 0", "Starting TrustedInstaller finalization.", "Ending the TrustedInstaller main loop.", "Idle processing thread terminated normally", "0000000e Created NT transaction (seq 2) result 0x00000000, handle @0xb8")
  → severity_text: 1 unique values (e.g., "Info")
  → attributes.custom.timestamp: 95 unique values (e.g., "2025 11 14 15:27:00", "2025 11 14 15:26:58", "2025 11 14 15:26:57", "2025 11 14 15:26:56", "2025 11 14 15:26:54", "2025 11 14 15:26:53", "2025 11 14 15:26:52", "2025 11 14 15:26:50", "2025 11 14 15:26:49", "2025 11 14 15:26:48")
  → resource.attributes.service.name: 2 unique values (e.g., "CBS", "CSI")
- logs.thunderbird-logs: 0.6190476190476191
  → field_10: 1 unique values (e.g., "")
  → body.text: 2 unique values (e.g., "opened for user root by (uid=0)", "closed for user root")
  → field_12: 1 unique values (e.g., "session")
  → attributes.host.hostname: 13 unique values (e.g., "dn754/dn754", "dn978/dn978", "en74/en74", "dn3/dn3", "dn261/dn261", "dn731/dn731", "src@eadmin1", "dn73/dn73", "dn228/dn228", "dn596/dn596")
  → attributes.custom.timestamp_text: 1 unique values (e.g., "2005.11.09 Nov 9 12:01:01")
  → attributes.process.name: 1 unique values (e.g., "crond")
  → resource.attributes.process.pid: 12 unique values (e.g., "2913", "2920", "3080", "2907", "2916", "4307", "2917", "2915", "2727", "12636")
  → attributes.custom.timestamp: 3 unique values (e.g., "1763134020", "1763134018", "1763134017")
  → attributes.user.name: 1 unique values (e.g., "pam_unix")
  → resource.attributes.host.name: 13 unique values (e.g., "dn754", "dn978", "en74", "dn3", "dn261", "dn731", "eadmin1", "dn73", "dn228", "dn596")
- logs.proxifier-logs: 1
  → attributes.event.type: 2 unique values (e.g., "open", "close,")
  → attributes.url.domain: 1 unique values (e.g., "proxy.cse.cuhk.edu.hk:5070")
  → attributes.custom.details: 38 unique values (e.g., "through proxy proxy.cse.cuhk.edu.hk:5070 HTTPS", "1190 bytes (1.16 KB) sent, 1671 bytes (1.63 KB) received, lifetime 00:02", "845 bytes sent, 12076 bytes (11.7 KB) received, lifetime <1 sec", "1165 bytes (1.13 KB) sent, 815 bytes received, lifetime <1 sec", "850 bytes sent, 10547 bytes (10.2 KB) received, lifetime 00:02", "0 bytes sent, 0 bytes received, lifetime <1 sec", "3425 bytes (3.34 KB) sent, 212164 bytes (207 KB) received, lifetime 00:18", "934 bytes sent, 5869 bytes (5.73 KB) received, lifetime <1 sec", "451 bytes sent, 18846 bytes (18.4 KB) received, lifetime <1 sec", "1293 bytes (1.26 KB) sent, 2439 bytes (2.38 KB) received, lifetime <1 sec")
  → attributes.custom.timestamp: 2 unique values (e.g., "11.14 15:27:01", "11.14 15:27:00")

Average Parsing Score (samples): 0.9577777777777778
Average Parsing Score (all docs): 0.9223184223184222
```


</details>

---------

Co-authored-by: kibanamachine <[email protected]>

Author: flash1293

.github/CODEOWNERS

@@ -930,6 +930,7 @@ x-pack/platform/packages/shared/kbn-apm-types @elastic/obs-ux-infra_services-tea
 x-pack/platform/packages/shared/kbn-classic-stream-flyout @elastic/kibana-management
 x-pack/platform/packages/shared/kbn-content-packs-schema @elastic/streams-program-team
 x-pack/platform/packages/shared/kbn-data-forge @elastic/obs-ux-management-team
+x-pack/platform/packages/shared/kbn-dissect-heuristics @elastic/obs-onboarding-team
 x-pack/platform/packages/shared/kbn-elastic-assistant @elastic/security-generative-ai
 x-pack/platform/packages/shared/kbn-elastic-assistant-common @elastic/security-generative-ai
 x-pack/platform/packages/shared/kbn-elastic-assistant-shared-state @elastic/security-generative-ai

package.json

@@ -530,6 +530,7 @@
     "@kbn/discover-plugin": "link:src/platform/plugins/shared/discover",
     "@kbn/discover-shared-plugin": "link:src/platform/plugins/shared/discover_shared",
     "@kbn/discover-utils": "link:src/platform/packages/shared/kbn-discover-utils",
+    "@kbn/dissect-heuristics": "link:x-pack/platform/packages/shared/kbn-dissect-heuristics",
     "@kbn/doc-links": "link:src/platform/packages/shared/kbn-doc-links",
     "@kbn/dom-drag-drop": "link:src/platform/packages/shared/kbn-dom-drag-drop",
     "@kbn/ebt-tools": "link:src/platform/packages/shared/kbn-ebt-tools",

tsconfig.base.json

@@ -922,6 +922,8 @@
       "@kbn/discover-shared-plugin/*": ["src/platform/plugins/shared/discover_shared/*"],
       "@kbn/discover-utils": ["src/platform/packages/shared/kbn-discover-utils"],
       "@kbn/discover-utils/*": ["src/platform/packages/shared/kbn-discover-utils/*"],
+      "@kbn/dissect-heuristics": ["x-pack/platform/packages/shared/kbn-dissect-heuristics"],
+      "@kbn/dissect-heuristics/*": ["x-pack/platform/packages/shared/kbn-dissect-heuristics/*"],
       "@kbn/doc-links": ["src/platform/packages/shared/kbn-doc-links"],
       "@kbn/doc-links/*": ["src/platform/packages/shared/kbn-doc-links/*"],
       "@kbn/docs-utils": ["packages/kbn-docs-utils"],

x-pack/platform/packages/shared/kbn-dissect-heuristics/README.md

@@ -0,0 +1,26 @@
+# @kbn/dissect-heuristics
+
+Utilities and helper functions for extracting Dissect patterns from log messages.
+
+## Overview
+
+This package provides an algorithm to automatically generate [Elasticsearch Dissect processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/dissect-processor.html) patterns by analyzing sample log messages. Unlike Grok patterns which use regular expressions, Dissect patterns use simple literal string delimiters for faster parsing of structured logs.
+
+## Supported Dissect Modifiers
+
+The extraction algorithm supports a subset of Dissect modifiers:
+
+- **Right Padding (`->`)**: Handles variable trailing whitespace
+- **Named Skip (`?`)**: Skips fields with non-meaningful constant values
+- **Empty Skip (`{}`)**: Anonymous skip fields
+
+**Note**: Reference keys (`*` and `&`) and append modifiers (`+`) are not supported by this implementation.
+
+## Delimiter Reliability Heuristics
+
+The current approach keeps delimiter scoring deliberately simple:
+
+- Position consistency: Delimiters are scored only by how consistently they appear at similar character offsets across messages (variance-based exponential decay).
+- Symmetry enforcement: After scoring, any single-character closing bracket `)`, `]`, `}` is discarded unless its matching opener `(`, `[`, `{` was also selected. This avoids generating patterns that fragment bracketed content using an orphan closer.
+
+No additional bracket penalties (mismatch, crossing, depth variance, ordering instability) are applied—favoring simpler, more predictable behavior while still preventing obviously broken delimiter choices.

x-pack/platform/packages/shared/kbn-dissect-heuristics/index.ts

@@ -0,0 +1,25 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+export { extractDissectPattern } from './src/extract_dissect_pattern';
+export { getDissectProcessor } from './src/get_dissect_processor';
+export { getReviewFields, type ReviewFields } from './src/review/get_review_fields';
+export { getDissectProcessorWithReview } from './src/review/get_dissect_processor_with_review';
+export { ReviewDissectFieldsPrompt } from './src/review/review_fields_prompt';
+export { groupMessagesByPattern } from './src/group_messages';
+export { serializeAST } from './src/serialize_ast';
+export type {
+  DissectPattern,
+  DissectField,
+  DissectModifiers,
+  DelimiterNode,
+  DissectProcessorResult,
+  DissectAST,
+  DissectASTNode,
+  DissectFieldNode,
+  DissectLiteralNode,
+} from './src/types';

x-pack/platform/packages/shared/kbn-dissect-heuristics/jest.config.js

@@ -0,0 +1,12 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+module.exports = {
+  preset: '@kbn/test/jest_node',
+  rootDir: '../../../../..',
+  roots: ['<rootDir>/x-pack/platform/packages/shared/kbn-dissect-heuristics'],
+};

x-pack/platform/packages/shared/kbn-dissect-heuristics/kibana.jsonc

@@ -0,0 +1,7 @@
+{
+  "type": "shared-common",
+  "id": "@kbn/dissect-heuristics",
+  "owner": "@elastic/obs-onboarding-team",
+  "group": "platform",
+  "visibility": "shared"
+}

x-pack/platform/packages/shared/kbn-dissect-heuristics/package.json

@@ -0,0 +1,6 @@
+{
+  "name": "@kbn/dissect-heuristics",
+  "private": true,
+  "version": "1.0.0",
+  "license": "Elastic License 2.0"
+}

x-pack/platform/packages/shared/kbn-dissect-heuristics/src/__fixtures__/log_samples.ts

@@ -0,0 +1,155 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+/**
+ * Apache Common Log Format
+ * Pattern: %{clientip} %{ident} %{auth} [%{timestamp}] "%{verb} %{request} %{httpversion}" %{response} %{bytes}
+ */
+export const APACHE_LOGS = [
+  '127.0.0.1 - - [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326',
+  '127.0.0.1 - - [10/Oct/2000:13:55:37 -0700] "GET /images/logo.png HTTP/1.0" 200 1234',
+  '192.168.1.1 - frank [10/Oct/2000:13:55:38 -0700] "POST /api/submit HTTP/1.1" 201 512',
+];
+
+/**
+ * Syslog format
+ * Pattern: %{month} %{day} %{time} %{hostname} %{process}[%{pid}]: %{message}
+ */
+export const SYSLOG = [
+  'Mar 10 15:45:23 hostname sshd[12345]: Accepted password for user from 192.168.1.1',
+  'Mar 10 15:45:24 hostname systemd[1]: Started service successfully',
+  'Mar 10 15:45:25 hostname kernel[0]: Memory allocation completed',
+];
+
+/**
+ * Custom pipe-delimited format
+ * Pattern: %{date}|%{level}|%{service}|%{message}
+ */
+export const PIPE_DELIMITED = [
+  '2024-01-15|INFO|UserService|User login successful',
+  '2024-01-15|WARN|AuthService|Rate limit approaching',
+  '2024-01-15|ERROR|DatabaseService|Connection timeout',
+];
+
+/**
+ * Logs with variable trailing whitespace (needs right padding modifier)
+ * Pattern: %{level->} %{message}
+ */
+export const VARIABLE_WHITESPACE = [
+  'INFO   Log message here',
+  'WARN Log message here',
+  'ERROR  Log message here',
+  'DEBUG    Log message here',
+];
+
+/**
+ * Logs with skip fields (constant values)
+ * Pattern: %{ip} %{} %{} [%{timestamp}]
+ */
+export const WITH_SKIP_FIELDS = [
+  '1.2.3.4 - - [30/Apr/1998:22:00:52]',
+  '5.6.7.8 - - [01/May/1998:10:15:30]',
+  '9.0.1.2 - - [02/May/1998:14:22:18]',
+];
+
+/**
+ * Simple space-delimited
+ * Pattern: %{field1} %{field2} %{field3}
+ */
+export const SPACE_DELIMITED = ['alpha beta gamma', 'one two three', 'red green blue'];
+
+/**
+ * JSON-like structure (challenging case)
+ * Pattern: {"timestamp":"%{timestamp}","level":"%{level}","message":"%{message}"}
+ */
+export const JSON_LIKE = [
+  '{"timestamp":"2024-01-15T10:30:00","level":"INFO","message":"User logged in"}',
+  '{"timestamp":"2024-01-15T10:30:01","level":"WARN","message":"High memory usage"}',
+  '{"timestamp":"2024-01-15T10:30:02","level":"ERROR","message":"Database error"}',
+];
+
+/**
+ * CSV format
+ * Pattern: %{id},%{name},%{email},%{status}
+ */
+export const CSV_FORMAT = [
+  '1,John Doe,[email protected],active',
+  '2,Jane Smith,[email protected],inactive',
+  '3,Bob Johnson,[email protected],active',
+];
+
+/**
+ * Edge case: No common delimiters
+ */
+export const NO_COMMON_DELIMITERS = ['abc123', 'xyz789', 'def456'];
+
+/**
+ * Edge case: All identical messages
+ */
+export const IDENTICAL_MESSAGES = [
+  'Same message every time',
+  'Same message every time',
+  'Same message every time',
+];
+
+/**
+ * Edge case: Single message
+ */
+export const SINGLE_MESSAGE = ['This is a single log message'];
+
+/**
+ * Edge case: Empty messages
+ */
+export const EMPTY_MESSAGES = ['', '', ''];
+
+/**
+ * Complex nested delimiters
+ * Pattern: %{date} [%{level}] (%{module}) - %{message}
+ */
+export const NESTED_DELIMITERS = [
+  '2024-01-15 [INFO] (UserService) - User authenticated successfully',
+  '2024-01-15 [WARN] (AuthService) - Invalid token detected',
+  '2024-01-15 [ERROR] (DatabaseService) - Query execution failed',
+];
+
+/**
+ * Multiple spaces as delimiters
+ * Pattern: %{field1}  %{field2}  %{field3}
+ */
+export const MULTIPLE_SPACES = [
+  'field1  field2  field3',
+  'data1  data2  data3',
+  'test1  test2  test3',
+];
+
+/**
+ * Tab-delimited format
+ * Pattern: %{col1}\t%{col2}\t%{col3}
+ */
+export const TAB_DELIMITED = ['col1\tcol2\tcol3', 'val1\tval2\tval3', 'data1\tdata2\tdata3'];
+
+/**
+ * Kubernetes-style logs
+ * Pattern: %{timestamp} %{stream} %{flag} %{message}
+ */
+export const KUBERNETES_LOGS = [
+  '2024-01-15T10:30:00.000Z stdout F Container started successfully',
+  '2024-01-15T10:30:01.000Z stderr F Error: Connection refused',
+  '2024-01-15T10:30:02.000Z stdout P Partial log line continues',
+];
+
+/**
+ * Expected patterns for testing
+ */
+export const EXPECTED_PATTERNS = {
+  APACHE_LOGS:
+    '%{clientip} %{ident} %{auth} [%{timestamp}] "%{verb} %{request} %{httpversion}" %{response} %{bytes}',
+  SYSLOG: '%{month} %{day} %{time} %{hostname} %{process}[%{pid}]: %{message}',
+  PIPE_DELIMITED: '%{date}|%{level}|%{service}|%{message}',
+  SPACE_DELIMITED: '%{field1} %{field2} %{field3}',
+  CSV_FORMAT: '%{id},%{name},%{email},%{status}',
+};