Chunking Strategies: Token-Based, Header-Based, Semantic, Overlap and Contextual Retrieval

Version note: The code in this post was built and run with .NET 10, Microsoft.Extensions.AI 10.11.0, Microsoft.ML.Tokenizers 2.0.1 and OllamaSharp 5.5.2 against local models (nomic-embed-text, gemma3:4b) on Ollama 0.34.4.

Guess First

You have the same 30 questions, the same embedding model and the same document set. The only thing you change is how you split the documents. Which of these two ranks the right section first more often: turning the section under every Markdown heading into one piece, or ignoring headings altogether and splitting the text into 64-token windows? Your intuition probably sides with headings. The measurement says something a little different, but the story is not a single number.

In the RAG (Retrieval-Augmented Generation) pipeline, chunking is the step that decides what the embedding model "sees". In the second post we saw that the vector of a long text loses any single sentence in the average; a very short piece, on the other hand, loses its context. In this post we measure four strategies and Anthropic's Contextual Retrieval method with the same question set.

Four Strategies, One Document

Fixed-size chunking splits text into equal pieces without knowing its structure; overlap repeats part of the neighboring pieces so that a sentence cut at a boundary survives in both. Header-based chunking trusts the document's own structure: every section is a chunk. Semantic chunking splits the text into sentences and starts a new chunk when the embeddings of two consecutive sentences drift apart; it tries to learn from the model where the topic changes. Parent-child (also called small-to-big) uses two sizes: it searches with small pieces (sentences here) and puts the big piece (the section that sentence belongs to) into the prompt.

Splitting by Counting Tokens

Fixed size has to be measured in tokens, not characters; the model's input limit and the prompt cost are measured in tokens. Microsoft.ML.Tokenizers gives us both the token count and each token's position in the text (here Tok is TiktokenTokenizer.CreateForEncoding("o200k_base")):

// Fixed size: windows of N tokens, stepping back by overlap tokens
public static List<SpannedChunk> FixedTokens(IEnumerable<FlatDoc> docs, int size, int overlap)
{
    List<SpannedChunk> result = [];
    foreach (FlatDoc f in docs)
    {
        IReadOnlyList<EncodedToken> tokens = Tok.EncodeToTokens(f.Text, out _);
        int n = 0;
        for (int i = 0; i < tokens.Count; i += size - overlap)
        {
            int j = Math.Min(i + size, tokens.Count);
            int start = tokens[i].Offset.Start.Value, end = tokens[j - 1].Offset.End.Value;
            result.Add(new(Chunkers.ToChunk(f.Doc, "", f.Text[start..end], n++), start, end));
            if (j == tokens.Count) break;
        }
    }
    return result;
}

Every chunk also carries its start and end character in the document's flattened text. The measurement uses this: a chunk counts as a "hit" if it overlaps the expected section. For the header-based chunker that is the same as the "right document, right section" rule we have used since the first post.

For header-based splitting, Microsoft.Extensions.DataIngestion ships HeaderChunker (we used it in the third post; MaxTokensPerChunk and OverlapTokens are set through IngestionChunkerOptions). In the measurement setup I wrote the same logic with our own types so that all four strategies produce the same span information.

Semantic and Parent-Child

For the semantic chunker I first split the documents into sentences. Since Turkish uses a dot as the thousands separator ("5.000 TL"), I defined a sentence end as "a period followed by whitespace or end of line"; table rows are units of their own. The 10 documents produced 161 sentences:

// Semantic: start a new chunk when the similarity of consecutive sentences drops below the threshold (unaware of headings)
public static async Task<List<SpannedChunk>> SemanticAsync(IEnumerable<FlatDoc> docs,
    IEmbeddingGenerator<string, Embedding<float>> embedder, float breakBelow)
{
    var sentences = Sentences(docs);
    GeneratedEmbeddings<Embedding<float>> vectors =
        await embedder.GenerateAsync(sentences.Select(s => "search_document: " + s.Text));
    List<SpannedChunk> result = [];
    int startIdx = 0, ordinal = 0;
    for (int i = 1; i <= sentences.Count; i++)
    {
        bool newDoc = i == sentences.Count || sentences[i].Doc != sentences[i - 1].Doc;
        bool topicShift = !newDoc && TensorPrimitives.CosineSimilarity(
            vectors[i - 1].Vector.Span, vectors[i].Vector.Span) < breakBelow;
        if (newDoc || topicShift)
        {
            var first = sentences[startIdx];
            var last = sentences[i - 1];
            string text = first.Doc.Text[first.Start..last.End];
            result.Add(new(Chunkers.ToChunk(first.Doc.Doc, "", text, ordinal++), first.Start, last.End));
            startIdx = i;
            if (newDoc) ordinal = 0;
        }
    }
    return result;
}

The threshold is the similarity two sentences need to count as "the same topic". Because nomic-embed-text's Turkish scores sit in a narrow band (in the first post the scores of right and wrong answers were between 0.65 and 0.87), I tried 0.75 and 0.70. Parent-child turns every sentence into its own chunk, prefixed with the document and section name; the search returns not the sentence but the section it belongs to:

// Parent-child: find with the small piece (sentence), return the big piece (section)
public static List<SpannedChunk> Children(IEnumerable<FlatDoc> docs) =>
    Sentences(docs).Select((s, i) => new SpannedChunk(
        Chunkers.ToChunk(s.Doc.Doc, s.Heading, $"{s.Doc.Doc.Title} > {s.Heading}: {s.Text}", i),
        s.Start, s.End, ParentId: $"{s.Doc.Doc.Id}::{s.Heading}")).ToList();

// parentMode: fetch 30 sentences, map them to their sections, drop duplicates, return the first 5 sections
if (parentMode)
{
    FlatDoc f = byId[sc.Chunk.DocId];
    var parent = f.Sections.First(s => s.Start <= sc.Start && sc.End <= s.End);
    if (results.Any(r => r.DocId == sc.Chunk.DocId && r.Start == parent.Start)) continue;
    results.Add((sc.Chunk.DocId, parent.Start, parent.End, f.Text[parent.Start..parent.End]));
}
else
{
    results.Add((sc.Chunk.DocId, sc.Start, sc.End, sc.Chunk.Content));
}

Measuring with the Same 30 Questions

$ dotnet run -- chunk-bench headers fixed64 fixed128 fixed128o fixed256 semantic semantic70 parent
header-based           chunks=  53 avgTok=   79  recall@1=16/30 recall@5=25/30 MRR=0.668  ctxTok@5= 399 sections@5=5.0
fixed 64, overlap 16   chunks=  82 avgTok=   60  recall@1=19/30 recall@5=25/30 MRR=0.717  ctxTok@5= 311 sections@5=4.4
fixed 128, overlap 0   chunks=  33 avgTok=  115  recall@1=19/30 recall@5=26/30 MRR=0.702  ctxTok@5= 567 sections@5=4.9
fixed 128, overlap 32  chunks=  40 avgTok=  119  recall@1=18/30 recall@5=24/30 MRR=0.682  ctxTok@5= 601 sections@5=4.9
fixed 256, overlap 0   chunks=  18 avgTok=  211  recall@1=17/30 recall@5=28/30 MRR=0.725  ctxTok@5=1122 sections@5=5.0
semantic (<0.75)       chunks= 110 avgTok=   31  recall@1=19/30 recall@5=24/30 MRR=0.701  ctxTok@5= 325 sections@5=4.8
semantic (<0.70)       chunks=  76 avgTok=   46  recall@1=20/30 recall@5=26/30 MRR=0.747  ctxTok@5= 667 sections@5=4.9
parent-child           chunks= 161 avgTok=   38  recall@1=16/30 recall@5=23/30 MRR=0.644  ctxTok@5= 372 sections@5=5.0

$ dotnet run -- chunk-bench contextual
contextualized 53 chunks in 298 s, input tokens=38272, output tokens=2443
header + contextual    chunks=  53 avgTok=  126  recall@1=18/30 recall@5=25/30 MRR=0.697  ctxTok@5= 647 sections@5=5.0

Columns: chunk count, average tokens per chunk, recall@1 and recall@5 (is the right section in the first 1 or 5 results), MRR (Mean Reciprocal Rank), the total tokens the first 5 results would put into the prompt, and how many distinct sections those 5 results cover.

First, the answer to the opening question: 64-token fixed windows ranked the right section first for 19 of 30 questions, the header-based chunker for 16. Part of the reason is the dilution we saw in the second post: the vector of a short piece looks more like the information it contains. But you need to keep three caveats in mind while reading the table.

First, the hit definition favors large chunks. A 256-token window covers two or three sections on average; touching the expected section is enough. That is why recall@5 rises to 28, and the price is on the same row: 1,122 tokens in the prompt, against 399 for header-based. If you do not read recall together with the context budget, you end up with the wrong conclusion that "big chunks are always better".

Second, a hit does not mean "the answer is inside the chunk". A fixed window may contain only the last sentence of a section and the start of the next; the measurement counts that as a hit, but the model may not find the answer. Tables are the clearest example:

$ dotnet run -- fixed-dump kargo-ve-teslimat 64 16
...
--- kargo-ve-teslimat#3 [489..712]
 ertesi iş günü teslim edilir. Ekspres teslimat ücreti sipariş tutarından bağımsız olarak 89,90 TL'dir ve ücretsiz kargo kampanyasına dahil değildir.

## Bölgelere göre teslim süreleri
| Bölge | Standart kargo | Ekspres |
|
--- kargo-ve-teslimat#4 [669..844]
leri
| Bölge | Standart kargo | Ekspres |
|---|---|---|
| İstanbul, Ankara, İzmir merkez | 1-2 iş günü | Ertesi iş günü |
| Diğer il merkezleri | 2-4 iş günü | Yok |
| İlçe ve
--- kargo-ve-teslimat#5 [801..957]
 merkezleri | 2-4 iş günü | Yok |
| İlçe ve köyler | 3-5 iş günü | Yok |
| KKTC | 5-8 iş günü | Yok |

## Yurt dışı gönderim
Şu an yalnızca KKTC'ye (Kuzey K

The row "İlçe ve köyler | 3-5 iş günü | Yok" (districts and villages | 3-5 business days | none) ended up in a chunk separate from the row with the column headers. A model reading this chunk cannot know whether "3-5 business days" refers to standard shipping or express. That is also why the question "how many days to a village in Van?" was lost in minimal RAG in the first post.

Third, 30 questions is a small set. One question is 3.3 points; the difference between 18 and 19 is noise. What I can say with confidence is coarser: 128 tokens with overlap is not better than 128 tokens without (recall@5 fell from 26 to 24, context grew from 567 to 601 tokens); parent-child gave the worst result on this set; semantic chunking (with the 0.70 threshold) gave the highest MRR, but its first 5 results average 667 tokens, about 1.7 times header-based.

Parent-child doing badly is interesting. A single Turkish sentence carries too little signal for an English-centric embedding model; the sentence "points cannot be converted to cash" is no closer to "can I turn my points into money?" than the whole section is. The method is not bad; for this model and this language a sentence is too small a search unit. Trying paragraphs or 2–3 sentence windows is the natural next step.

Contextual Retrieval

Anthropic's Contextual Retrieval method, published on 19 September 2024, suggests this: before embedding each chunk, have an LLM (Large Language Model) that sees the whole document write 50–100 tokens of context describing the chunk's place in the document, and prepend it to the chunk. The reported result is a reduction in retrieval failures in the top 20 results of 35% with contextual embeddings alone, 49% together with contextual BM25 (Best Matching 25, a classic keyword ranking algorithm), and 67% once reranking is added. The same article also suggests putting everything in the prompt instead of RAG when the knowledge base is under 200,000 tokens; our 4,184-token set is far below that limit, but it is a good lab for measuring the method.

For every header-based chunk I gave gemma3:4b the whole document and the chunk:

ChatResponse response = await chat.GetResponseAsync(
[
    new(ChatRole.System, "You write short retrieval context for document chunks. Reply in Turkish with one or two sentences and nothing else."),
    new(ChatRole.User, $"""
        <document>
        {byId[c.Chunk.DocId].Text}
        </document>
        <chunk>
        {c.Chunk.Content}
        </chunk>
        Write one or two sentences that place this chunk within the document (which document, which topic, what it applies to), so that the chunk can be found by search.
        """)
], new ChatOptions { Temperature = 0, Seed = 42 });

The result is the last row of the measurement table: recall@1 went from 16 to 18, MRR from 0.668 to 0.697, recall@5 stayed at 25. 53 chunks cost 298 seconds and 38,272 input tokens; because every call reads the whole document again, the input cost grows not with the chunk count but with "chunk count × document length". That is why Anthropic's cost estimate relies on prompt caching; a local model does not have that advantage.

The reason the gain stayed small becomes clear when you look at the generated contexts. For the first section of the shipping policy the model wrote "Bu bölüm, standart kargo ile ilgili teslimat süreleri ve ücretlendirme bilgilerini açıklamaktadır." ("This section explains delivery times and pricing for standard shipping."). That sentence restates the section heading; our chunks already started with "Kargo ve Teslimat > Standart kargo". Contextual Retrieval helps most with headless chunks cut off from their context; if you already prepend the heading path, you have taken most of the gain the cheap way.

Production Notes

Do not pick a chunk size once and forget it. "Start around a few hundred tokens and tune by measuring" is a good default; on this set the best results came from 30–120 tokens, but that holds for short Turkish policy texts. For long technical documents the balance may be elsewhere.

Choose the chunking strategy by document type. In content that is already sectioned, such as policies and FAQs (Frequently Asked Questions), header-based chunking keeps the natural boundaries; for long text without headings (contract annexes, email threads) fixed windows or semantic chunking fit better. Set a separate rule for tables and code blocks: either one whole chunk, or every row with the header row prepended.

Changing chunking is a reindexing operation. Every document is re-chunked and re-embedded, and thresholds shift (in the first post we saw the 0.70 threshold eliminate everything when the chunker changed). Validate the change with the measurement set and roll it out with the "new column, backfill, switch" sequence from the second post.

Add the context budget to the measurement table. Every change that raises recall may also grow the prompt; in the tenth post we manage that budget directly.

Common Pitfalls

First, splitting a table in the middle. A table row separated from its column headers is a meaningless string of numbers to the model. Do not cut sections containing tables with fixed windows, or repeat the header row in every piece.

Second, raising overlap and filling the context with duplicate results. On this set 32 tokens of overlap did not raise recall; it grew the context. Overlap's benefit is rescuing the sentence cut at a boundary, but if two neighboring chunks come back for the same question, the same text enters the prompt twice.

Third, searching for a single "best chunk size". The best by recall@1 is not the best by recall@5 or by context budget, and on a 30-question set rankings change with one or two questions. Compare a few reasonable options with your own question set and more than one metric.

Fourth, not questioning the hit definition. The "chunk overlaps the section" rule favors big chunks. For a stricter measurement, record the expected answer sentence for each question and check whether the chunk fully contains it; we cover that in the twelfth post.

Fifth, turning on Contextual Retrieval without measuring. It raises ingestion cost by chunk count × document length; if the heading path is already in the chunk, the gain may stay small.

A Small Experiment

★☆☆ Add a 32-token fixed window (overlap 8) to chunk-bench. Does recall@1 keep rising, or do the chunks start losing their meaning?

★★☆ Change Children to work with 2-sentence windows (1 sentence overlap) instead of single sentences. Does parent-child's recall@5 beat header-based?

★★★ Write a rule that protects table rows in fixed windows: if a window starts inside a table, prepend the table's header row to the chunk. Compare the chunk retrieved for "Siparişim Van'ın bir köyüne kaç günde gelir?" (how many days to a village in Van?) and the model's answer before and after.

Our chunks are ready; now it is time to decide where to store them. In the next post we look at vector database options, exact and approximate search, HNSW (Hierarchical Navigable Small World) and IVFFlat, and how filters behave together with ANN (Approximate Nearest Neighbor) search.