[Question]: How to chunk a Markdown file only by a custom delimiter (e.g., <hr>) without further splitting?

[VVX94]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (Language Policy).
• Non-english title submitions will be closed directly ( 非英文标题的提交将会被直接关闭 ) (Language Policy).
• Please do not modify this template :) and fill in all the required fields.

Describe your problem

My goal

I have some Markdown (.md) files that I have pre-processed. Each entry/record in this file is explicitly separated by \n\n<hr>\n\n.

My goal is to have RAGFlow treat each section separated by <hr> as a single, complete chunk, regardless of how long that section is.

Example data structure in my_file.md:

# Entry 1 Title
This is the content for entry 1.
It might be very long, over 1000 characters.
## SubTitle
...
...

<hr>

# Entry 2 Title
This is the content for entry 2.
It might be short.

<hr>

# Entry 3 Title
This is another long entry.
...

And my desired chunks should be Chunk 1: (All content for Entry 1),Chunk 2: (All content for Entry 2)...

What I have tried

Using the "General" Knowledge Base Import

When I upload the file directly to a Knowledge Base using the "general" method, RAGFlow still splits my entries. For example, if "Entry 1" is 1500 characters and my chunk_size is 512, though I set "Text segmentation identifier" as \n\n<hr>\n\n,RAGFlow splits "Entry 1" into 3-4 smaller chunks. This breaks the context of my data.(in this case,I have 1000 structured entries, and chunk_size is 512, my file is split into 2127)

Using an Ingestion Pipeline (Agent):

I tried to build a custom processing pipeline to control this. My pipeline looks like this(the template): File -> Parser -> Chunker -> Tokenizer (Indexer)

I hope the chunker can split the markdown file with H1, while I come across an error[ERROR]Input error: ... Input should be 'json' or 'chunks' [type=literal_error, input_value='text']

My Question

How can I correctly configure RAGFlow to chunk only on my custom <hr> separator and prevent any further splitting (like by chunk_size or paragraphs) within those sections?
A) Is there a setting in the standard Knowledge Base import that I missed (e.g., "split by custom regex" or "disable size limit")?

B) If this must be done with an Ingestion Pipeline, what is the correct node setup and configuration to achieve this? Is there anything I missed?

Thanks for your time : )

[dosubot]

RAGFlow's standard Knowledge Base import does not support chunking Markdown files solely by a custom delimiter like <hr> without further splitting—chunk size limits always apply, so long sections get split regardless of your delimiter setting source. There isn't a built-in option to disable size-based splitting or to split only by custom regex in the standard import source.

To achieve pure delimiter-based chunking, you need a custom ingestion pipeline. The recommended setup is: File → Parser → StringTransform → Mapper → Tokenizer/Indexer. Use the StringTransform component with the "split" method and set your delimiter to <hr> (or whatever you need). This component splits strictly on your delimiter and does not enforce chunk size limits source. You may need a Mapper node to convert the output to the expected chunk format for downstream nodes.

If you encountered a pipeline error about input format (e.g., "Input should be 'json' or 'chunks', but received 'text'"), make sure each node's output matches the next node's expected input. For chunkers, the input should be in 'chunks' or 'json' format, not raw 'text' source.

In summary: use a custom pipeline with StringTransform for delimiter-only chunking, and ensure node input/output formats are compatible. Standard import cannot do this.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[buua436]

@VVX94
You’re right—the markdown parser previously prioritized recommended chunk size over delimiter for text, which made custom delimiters less effective. We’ve optimized this: if your delimiter is wrapped in backticks (e.g., `\n\n<hr>\n\n`), the parser now splits on that first; if no delimiter is provided, it falls back to document-structure splitting; after splitting, chunks are merged according to recommended chunk size. To meet your need, set delimiter for text to `\n\n<hr>\n\n` and choose a small recommended chunk size.
This improvement will roll out to the nightly build shortly,please wait a little moment.

[VVX94]

@buua436
Thank you for the quick reply and for optimizing the Knowledge Base parser! This fix sounds perfect for my use case (Part A of my question). I will test it soon!

I still have a follow-up question regarding Part B of my original issue, which was about the Ingestion Pipeline.

My pipeline (File -> Parser -> HierarchicalMerger) failed with this specific error: [ERROR]Input error: ... Input should be 'json' or 'chunks' [type=literal_error, input_value='text']

I'm trying to understand why this failed, as this seems to be the recommended pipeline for chunking.

Is it expected that the Parser node outputs text format for Markdown files?

I'd love to understand the correct way to build this pipeline, even if the Knowledge Base import is now fixed. Thanks again!

[buua436]

@VVX94
Sorry for the late reply. For Part B, you need to insert a token-based chunker between the parser and the title-based chunker.

[yiqiao123]

@VVX94 Sorry for the late reply. For Part B, you need to insert a token-based chunker between the parser and the title-based chunker.

Oh, it works for me, that is great, thanks!

But there is another problem, the vector simlarity seems not correct. I use this chunker and test retrieval with the text of one chunk, the vector simlarity of the same chunk in the search result is 23.27.
And I use the "general" chunking method to ingest the file again, the same chunk's vector simlarity is 90.94.

I set the chunker as shown in the figure below, "Filename embedding weight" in the indexer is set to 0. Is there anything I missed?

Looking forward to your reply, thanks very much.

[VVX94]

@VVX94 Sorry for the late reply. For Part B, you need to insert a token-based chunker between the parser and the title-based chunker.

Thanks a lot! It works for me too.

[buua436]

@VVX94 Sorry for the late reply. For Part B, you need to insert a token-based chunker between the parser and the title-based chunker.

Oh, it works for me, that is great, thanks!

But there is another problem, the vector simlarity seems not correct. I use this chunker and test retrieval with the text of one chunk, the vector simlarity of the same chunk in the search result is 23.27. And I use the "general" chunking method to ingest the file again, the same chunk's vector simlarity is 90.94.

I set the chunker as shown in the figure below, "Filename embedding weight" in the indexer is set to 0. Is there anything I missed?

Looking forward to your reply, thanks very much.

@yiqiao123 Thanks for reporting—this issue did exist and has now been fixed. Please refer to PR #10964 for the code changes and details.