feat: implement MarkdownHeaderLevelsInferrer (#373)

* add component and tests

* rework to match component-level output pattern

* improved test cases

* fix linting error

* resolve typing issues

* remove pytest from global dependencies

* move logger to top of file

* Update haystack_experimental/components/preprocessors/md_header_level_inferrer.py

Co-authored-by: David S. Batista <[email protected]>

* use doc.content instead of extra variable 'content'

* Update haystack_experimental/components/preprocessors/md_header_level_inferrer.py

Co-authored-by: David S. Batista <[email protected]>

* Update haystack_experimental/components/preprocessors/md_header_level_inferrer.py

Co-authored-by: David S. Batista <[email protected]>

* refactor for readability

* adding docstrings and simplyfing

* removing uv.lock

* adding new component to README.md

* extending tests

* adding link to discussion

---------

Co-authored-by: David S. Batista <[email protected]>

Author: OGuggenbuehl

README.md

@@ -41,16 +41,17 @@ that includes it. Once it reaches the end of its lifespan, the experiment will b
 
 ### Active experiments
 
-| Name                                  | Type                           | Expected End Date | Dependencies | Cookbook                                                                                                                                                                                                                                                  | Discussion    |
-|---------------------------------------|--------------------------------|-------------------|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
-| [`InMemoryChatMessageStore`][1]       | Memory Store                   | December 2024     | None         | <a href="https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/conversational_rag_using_memory.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>      | [Discuss][4]  |
-| [`ChatMessageRetriever`][2]           | Memory Component               | December 2024     | None         | <a href="https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/conversational_rag_using_memory.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>      | [Discuss][4]  |
-| [`ChatMessageWriter`][3]              | Memory Component               | December 2024     | None         | <a href="https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/conversational_rag_using_memory.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>      | [Discuss][4]  |
+| Name                                  | Type                           | Expected End Date | Dependencies | Cookbook                                                                                                                                                                                                                                            | Discussion    |
+|---------------------------------------|--------------------------------|-------------------|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
+| [`InMemoryChatMessageStore`][1]       | Memory Store                   | December 2024     | None         | <a href="https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/conversational_rag_using_memory.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> | [Discuss][4]  |
+| [`ChatMessageRetriever`][2]           | Memory Component               | December 2024     | None         | <a href="https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/conversational_rag_using_memory.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> | [Discuss][4]  |
+| [`ChatMessageWriter`][3]              | Memory Component               | December 2024     | None         | <a href="https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/conversational_rag_using_memory.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> | [Discuss][4]  |
 | [`QueryExpander`][5]                  | Query Expansion Component      | October 2025      | None         | None | [Discuss][6]  |
 | [`EmbeddingBasedDocumentSplitter`][8] | EmbeddingBasedDocumentSplitter | August 2025       | None         | None | [Discuss][7]  |
 | [`MultiQueryEmbeddingRetriever`][13]  | MultiQueryEmbeddingRetriever   | November 2025     | None         | None | [Discuss][11] |
 | [`MultiQueryTextRetriever`][14]       | MultiQueryTextRetriever        | November 2025     | None         | None | [Discuss][12] |
 | [`OpenAIChatGenerator`][9]            | Chat Generator Component       | November 2025     | None         | <a href="https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/hallucination_score_calculator.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/> | [Discuss][10] |
+| [`MarkdownHeaderLevelInferrer`][15]   | Preprocessor                   | January 2025      | None         | None                                                                                                                                                                                                                                                | [Discuss][16] |
 
 [1]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/chat_message_stores/in_memory.py
 [2]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/retrievers/chat_message_retriever.py
@@ -66,6 +67,10 @@ that includes it. Once it reaches the end of its lifespan, the experiment will b
 [12]: https://github.com/deepset-ai/haystack-experimental/discussions/364
 [13]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/retrievers/multi_query_embedding_retriever.py
 [14]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/retrievers/multi_query_text_retriever.py
+[15]: https://github.com/deepset-ai/haystack-experimental/blob/main/haystack_experimental/components/retrievers/md_header_level_inferrer.py
+[16]: https://github.com/deepset-ai/haystack-experimental/discussions/376
+
+
 
 ### Adopted experiments
 | Name                                                                                   | Type                                     | Final release |

haystack_experimental/components/preprocessors/md_header_level_inferrer.py

@@ -0,0 +1,146 @@
+# SPDX-FileCopyrightText: 2022-present deepset GmbH <[email protected]>
+#
+# SPDX-License-Identifier: Apache-2.0
+
+import re
+
+from haystack import Document, component, logging
+
+logger = logging.getLogger(__name__)
+
+
+@component
+class MarkdownHeaderLevelInferrer:
+    """
+    Infers and rewrites header levels in Markdown text to normalize hierarchy.
+
+    First header → Always becomes level 1 (#)
+    Subsequent headers → Level increases if no content between headers, stays same if content exists
+    Maximum level → Capped at 6 (######)
+
+    ### Usage example
+    ```python
+    from haystack import Document
+    from haystack_experimental.components.preprocessors import MarkdownHeaderLevelInferrer
+
+    # Create a document with uniform header levels
+    text = "## Title\nSome content\n## Section\nMore content\n## Subsection\nFinal content"
+    doc = Document(content=text)
+
+    # Initialize the inferrer and process the document
+    inferrer = MarkdownHeaderLevelInferrer()
+    result = inferrer.run([doc])
+
+    # The headers are now normalized with proper hierarchy
+    print(result["documents"][0].content)
+    > # Title\nSome content\n## Section\nMore content\n### Subsection\nFinal content
+    ```
+    """
+
+    def __init__(self):
+        """Initializes the MarkdownHeaderLevelInferrer."""
+        # handles headers with optional trailing spaces and empty content
+        self._header_pattern = re.compile(r"(?m)^(#{1,6})\s+(.+?)(?:\s*)$")
+
+    @component.output_types(documents=list[Document])
+    def run(self, documents: list[Document]) -> dict:
+        """
+        Infers and rewrites the header levels in the content for documents that use uniform header levels.
+
+        :param documents: list of Document objects to process.
+
+        :returns:
+            dict: a dictionary with the key 'documents' containing the processed Document objects.
+        """
+        if not documents:
+            logger.warning("No documents provided to process")
+            return {"documents": []}
+
+        logger.debug(f"Inferring and rewriting header levels for {len(documents)} documents")
+        processed_docs = [self._process_document(doc) for doc in documents]
+        return {"documents": processed_docs}
+
+    def _process_document(self, doc: Document) -> Document:
+        """
+        Processes a single document, inferring and rewriting header levels.
+
+        :param doc: Document object to process.
+        :returns:
+            Document object with rewritten header levels.
+        """
+        if doc.content is None:
+            logger.warning(f"Document {getattr(doc, 'id', '')} content is None; skipping header level inference.")
+            return doc
+
+        matches = list(re.finditer(self._header_pattern, doc.content))
+        if not matches:
+            logger.info(f"No headers found in document {doc.id}; skipping header level inference.")
+            return doc
+
+        modified_text = MarkdownHeaderLevelInferrer._rewrite_headers(doc.content, matches)
+        logger.info(f"Rewrote {len(matches)} headers with inferred levels in document{doc.id}.")
+        return MarkdownHeaderLevelInferrer._build_final_document(doc, modified_text)
+
+    @staticmethod
+    def _rewrite_headers(content: str, matches: list[re.Match]) -> str:
+        """
+        Rewrites the headers in the content with inferred levels.
+
+        :param content: Original Markdown content.
+        :param matches: List of regex matches for headers.
+        """
+        modified_text = content
+        offset = 0
+        current_level = 1
+
+        for i, match in enumerate(matches):
+            original_header = match.group(0)
+            header_text = match.group(2).strip()
+
+            # Skip empty headers
+            if not header_text:
+                logger.warning(f"Skipping empty header at position {match.start()}")
+                continue
+
+            has_content = MarkdownHeaderLevelInferrer._has_content_between_headers(content, matches, i)
+            inferred_level = MarkdownHeaderLevelInferrer._infer_level(i, current_level, has_content)
+            current_level = inferred_level
+
+            new_header = f"{'#' * inferred_level} {header_text}"
+            start_pos = match.start() + offset
+            end_pos = match.end() + offset
+            modified_text = modified_text[:start_pos] + new_header + modified_text[end_pos:]
+            offset += len(new_header) - len(original_header)
+
+        return modified_text
+
+    @staticmethod
+    def _has_content_between_headers(content: str, matches: list[re.Match], i: int) -> bool:
+        """Checks if there is content between the previous and current header."""
+        if i == 0:
+            return False
+        prev_end = matches[i - 1].end()
+        current_start = matches[i].start()
+        content_between = content[prev_end:current_start]
+        return bool(content_between.strip())
+
+    @staticmethod
+    def _infer_level(i: int, current_level: int, has_content: bool) -> int:
+        """Infers the header level for the current header."""
+        if i == 0:
+            return 1
+        if has_content:
+            return current_level
+        return min(current_level + 1, 6)
+
+    @staticmethod
+    def _build_final_document(doc: Document, new_content: str) -> Document:
+        """Creates a new Document with updated content, preserving other fields."""
+        return Document(
+            id=getattr(doc, "id", "") or "",
+            content=new_content,
+            blob=getattr(doc, "blob", None),
+            meta=getattr(doc, "meta", {}) or {},
+            score=getattr(doc, "score", None),
+            embedding=getattr(doc, "embedding", None),
+        )

test/components/preprocessors/test_markdown_header_level_inferrer.py

@@ -0,0 +1,162 @@
+# SPDX-FileCopyrightText: 2022-present deepset GmbH <[email protected]>
+#
+# SPDX-License-Identifier: Apache-2.0
+
+from haystack import Document
+from haystack_experimental.components.preprocessors.md_header_level_inferrer import MarkdownHeaderLevelInferrer
+
+
+def test_single_header_level_inference():
+    text = "## H1\nSome content\n## H2\nContent"
+    inferrer = MarkdownHeaderLevelInferrer()
+    doc = Document(content=text)
+    result = inferrer.run([doc])
+    content = result["documents"][0].content
+    # Expect the first header to be rewritten to level 1, second to level 1 (since content follows)
+    expected = "# H1\nSome content\n# H2\nContent"
+    assert content == expected
+
+
+def test_header_level_increase_on_consecutive_headers():
+    text = "## H1\n## H2\n## H3"
+    inferrer = MarkdownHeaderLevelInferrer()
+    doc = Document(content=text)
+    result = inferrer.run([doc])
+    content = result["documents"][0].content
+    # Expect the first header to be level 1, the next two to increase in level
+    expected = "# H1\n## H2\n### H3"
+    assert content == expected
+
+
+def test_no_headers():
+    text = "This is just some text without headers."
+    inferrer = MarkdownHeaderLevelInferrer()
+    doc = Document(content=text)
+    result = inferrer.run([doc])
+    content = result["documents"][0].content
+    assert content == text
+
+
+def test_complex_structure():
+    text = (
+        "## Title\n"
+        "## Section\n"
+        "Section content\n"
+        "## Subsection\n"
+        "Subsection content\n"
+        "## Another Section\n"
+        "## Another Subsection\n"
+        "Even more content\n"
+        "## Final Section\n"
+        "Final content"
+    )
+    inferrer = MarkdownHeaderLevelInferrer()
+    doc = Document(content=text)
+    result = inferrer.run([doc])
+    content = result["documents"][0].content
+    expected = (
+        "# Title\n"
+        "## Section\n"
+        "Section content\n"
+        "## Subsection\n"
+        "Subsection content\n"
+        "## Another Section\n"
+        "### Another Subsection\n"
+        "Even more content\n"
+        "### Final Section\n"
+        "Final content"
+    )
+    assert content == expected
+
+
+def test_empty_documents_list():
+    inferrer = MarkdownHeaderLevelInferrer()
+    result = inferrer.run([])
+    assert result["documents"] == []
+
+
+def test_document_with_none_content():
+    inferrer = MarkdownHeaderLevelInferrer()
+    doc = Document(content=None)
+    result = inferrer.run([doc])
+    assert result["documents"][0].content is None
+
+
+def test_document_with_empty_content():
+    inferrer = MarkdownHeaderLevelInferrer()
+    doc = Document(content="")
+    result = inferrer.run([doc])
+    assert result["documents"][0].content == ""
+
+
+def test_headers_with_trailing_spaces():
+    text = "## Header 1   \nContent\n## Header 2   \nMore content"
+    inferrer = MarkdownHeaderLevelInferrer()
+    doc = Document(content=text)
+    result = inferrer.run([doc])
+    content = result["documents"][0].content
+    expected = "# Header 1\nContent\n# Header 2\nMore content"
+    assert content == expected
+
+
+def test_headers_with_leading_spaces():
+    text = "  ## Header 1\nContent\n  ## Header 2\nMore content"
+    inferrer = MarkdownHeaderLevelInferrer()
+    doc = Document(content=text)
+    result = inferrer.run([doc])
+    # Headers with leading spaces should not match the pattern
+    assert result["documents"][0].content == text
+
+
+def test_maximum_header_level():
+    text = "## H1\n## H2\n## H3\n## H4\n## H5\n## H6\n## H7\n## H8"
+    inferrer = MarkdownHeaderLevelInferrer()
+    doc = Document(content=text)
+    result = inferrer.run([doc])
+    content = result["documents"][0].content
+    expected = "# H1\n## H2\n### H3\n#### H4\n##### H5\n###### H6\n###### H7\n###### H8"
+    assert content == expected
+
+
+def test_multiple_documents():
+    text1 = "## Title 1\nContent 1"
+    text2 = "## Title 2\nContent 2"
+    inferrer = MarkdownHeaderLevelInferrer()
+    docs = [Document(content=text1), Document(content=text2)]
+    result = inferrer.run(docs)
+    
+    assert len(result["documents"]) == 2
+    assert result["documents"][0].content == "# Title 1\nContent 1"
+    assert result["documents"][1].content == "# Title 2\nContent 2"
+
+
+def test_headers_with_special_characters():
+    text = "## Header with émojis 🚀\nContent\n## Header with numbers 123\nMore content"
+    inferrer = MarkdownHeaderLevelInferrer()
+    doc = Document(content=text)
+    result = inferrer.run([doc])
+    expected = "# Header with émojis 🚀\nContent\n# Header with numbers 123\nMore content"
+    assert result["documents"][0].content == expected
+
+
+def test_headers_with_markdown_formatting():
+    text = "## Header with **bold** text\nContent\n## Header with *italic* text\nMore content"
+    inferrer = MarkdownHeaderLevelInferrer()
+    doc = Document(content=text)
+    result = inferrer.run([doc])
+    expected = "# Header with **bold** text\nContent\n# Header with *italic* text\nMore content"
+    assert result["documents"][0].content == expected
+
+
+def test_very_long_content():
+    lines = ["## Header " + str(i) + "\nContent for header " + str(i) for i in range(50)]
+    text = "\n".join(lines)
+    inferrer = MarkdownHeaderLevelInferrer()
+    doc = Document(content=text)
+    result = inferrer.run([doc])
+    
+    # verify first header becomes level 1, others follow the pattern
+    content = result["documents"][0].content
+    assert content.startswith("# Header 0")
+    assert "# Header 1" in content
+    assert len(content.split("\n")) == len(text.split("\n"))