Your chunk size is the single biggest knob in your retrieval quality, and the default value you almost certainly inherited from a tutorial is wrong for your corpus. The chunks decide what the retriever ever sees. A chunk that cuts mid-sentence, mid-list, or mid-code-block carries half the semantic signal of the original passage, and a chunk that bundles five unrelated topics has none of the precision the LLM needs. Issue 006 covered hybrid retrieval to catch the queries pure vector search misses; this issue is one layer further upstream, where the chunks are produced in the first place.

The chunking decision is unusually load-bearing for the rest of the RAG stack. Embedding models and rerankers operate on the units of text the chunker creates; the LLM sees the chunks the retriever returns. If the chunks are misaligned with the semantic units of the source documents, no amount of better retrieval, reranking, or prompting recovers what was lost at indexing time. Get chunking right and every layer downstream has good raw material to work with. This issue walks through the three families of chunking strategy that dominate production, the tradeoffs each makes, and how to pick.

Why naive chunking destroys retrieval quality

Most RAG systems start with a fixed-size token chunker copied from a tutorial: 1000 tokens per chunk, no overlap, default tokeniser. The trouble shows up the first time a real document goes through it.

Consider a technical document with a heading, a paragraph of context, a numbered list of steps, and a code block. The default chunker counts to 1000 tokens, cuts wherever it lands (often in the middle of a list item or a code line), and starts the next chunk where it left off.

Three failure modes follow from that single design choice.

First, semantic units fragment. A list item that explained step three of a setup procedure is split across two chunks, and neither chunk on its own conveys what step three does. When the user searches for "step three setup", neither half-chunk scores high enough to retrieve, and the right answer simply does not surface.

Second, context disappears at the boundary. The first sentence of a paragraph often refers to something defined in the previous paragraph ("This approach has three weaknesses..."). When the chunk boundary cuts before "This", the reader of the chunk has no idea what "this" refers to. The embedding model loses signal at the boundary; the LLM, if it does receive the chunk, generates plausible-sounding nonsense around the missing antecedent.

Third, code blocks break in the worst possible way. A chunker that splits at a token count slices through code mid-line, mid-string, or mid-function. The retrieved chunk shows a fragment that compiles to nothing and reads as gibberish to a developer who finds it in the LLM's response. The brand cost of a single such response in a developer-facing product is significant.

The combined effect is a retrieval-quality ceiling no amount of downstream tuning can break through. Better embeddings, better rerankers, more candidates retrieved per query: none of them help if the chunks themselves are misaligned with the semantic units of the underlying documents.

Three families of chunking strategy

Three strategy families dominate production usage, presented below in roughly the order of increasing complexity. Most teams start with the first, move to the second when they encounter mixed document types, and adopt the third when their corpus mixes long-form context with precise retrieval queries.

Family one: fixed-size token chunking. Fixed-size chunking splits the document into equal-token windows with a configurable overlap. The chunker counts tokens (typically using the embedding model's own tokeniser), emits a chunk when the count reaches the configured size, and steps forward by the chunk size minus the overlap. A typical configuration is 512 tokens per chunk with 50 tokens of overlap. The overlap exists so that a sentence cut at a chunk boundary appears in both chunks, recovering some of the context lost at the cut. The strategy is simple, deterministic, fast, and language-agnostic, which makes it the right starting point for a uniform corpus of natural-language documents of similar length. It is the wrong starting point for a corpus that mixes prose, code, tables, and structured documents, because none of those have natural boundaries at fixed token counts.

Family two: structure-aware (recursive) chunking. Structure-aware chunking respects the natural boundaries of the document: headings, paragraphs, sentences, list items, code blocks. The popular implementation is LangChain's RecursiveCharacterTextSplitter, which tries to split on a hierarchy of separators (double newlines, single newlines, sentence boundaries, then character boundaries) and only falls back to the next level when the current level produces chunks too large for the configured limit. The strategy preserves semantic units at the cost of variable chunk size; a long paragraph in a technical document might produce a 700-token chunk, while a short paragraph might produce a 100-token chunk. The retriever then ranks variable-length chunks against each other, which most embedding models handle gracefully provided the variance is not extreme. Structure-aware chunking is the default recommendation for any corpus with markdown or HTML structure, and it outperforms fixed-size chunking on heterogeneous corpora while matching it on uniform-prose corpora.

Family three: hierarchical (parent-child) chunking. Hierarchical chunking maintains two levels of granularity. Small chunks (typically 100 to 200 tokens, a few sentences each) are embedded and indexed for retrieval. Each small chunk carries a reference to a larger parent (typically 500 to 1000 tokens, a paragraph or section), which is returned to the LLM in place of the small chunk after retrieval. The pattern decouples the retrieval unit from the generation unit. The retriever performs precise matching against short, focused chunks, where embedding models do their best work. The LLM then receives the larger parent, which carries enough surrounding context to actually generate a useful answer. A common variant is the sentence-window pattern: the small chunk is a single sentence, and the parent is that sentence plus three to five sentences on either side. The cost of hierarchical chunking is operational: the indexing pipeline maintains two collections, the retrieval pipeline performs a join from small chunk to parent at query time, and storage roughly doubles because the parent text is held alongside the small-chunk text.

The diagram above maps the three families onto the indexing pipeline. Fixed-size and structure-aware chunking produce a single collection; hierarchical chunking produces two collections that get joined at query time. The choice of family determines what the rest of the retrieval stack operates on, and it is the cheapest decision to get right at indexing time and the most expensive to change later, because changing strategy means re-chunking and re-embedding the entire corpus.

How to pick

The decision tree depends on three properties of the corpus and the application: document heterogeneity, latency budget, and the precision-versus-context tradeoff in the generation step.

For a uniform corpus of natural-language documents (FAQ entries, marketing pages, blog posts of similar length), fixed-size chunking at 512 tokens with 50-token overlap is the right starting point. It is the cheapest to operate, and the homogeneity of the documents means structure-aware splitters offer little measurable benefit.

For a heterogeneous corpus that mixes prose, markdown, code, and tables (technical documentation, internal wikis, GitHub repositories), structure-aware chunking is the right default. The semantic-unit preservation pays for itself within the first hundred queries that involve a code identifier or a list item, because the retriever now sees those identifiers as part of complete chunks rather than as fragments straddling a boundary.

For an application where retrieval precision is the binding constraint and the LLM has a generous context budget, hierarchical chunking is worth the operational cost. The pattern is particularly strong for question-answering over long technical documents, where the right answer lives in a single sentence but needs surrounding context to be useful to the reader.

A second, smaller decision tree governs the chunk size itself. Embedding models have an effective context limit, ranging from around 512 tokens for many older models (BGE-base and the original sentence-transformer family) up to 8192 tokens for modern models like bge-m3. Chunks larger than the model's effective context get truncated silently and underperform retrieval. Chunks much smaller than the effective context underuse the model's capacity and waste storage. The right size is the upper end of the effective context, less the overlap, less a small buffer for tokeniser differences between the chunker and the embedding model.

Two further design points are worth setting once and revisiting rarely. First, the separator hierarchy in a structure-aware splitter (for example, RecursiveCharacterTextSplitter's default ["\n\n", "\n", ". ", " ", ""]) is the entire semantic-preservation mechanism; reordering it carelessly collapses the structure-aware chunker into a fixed-size one. Second, every chunk should carry doc_id, chunk_idx, and a source page or section reference alongside the text. Most production RAG systems need both: doc_id to attribute the answer back to a source document, and chunk_idx to reconstruct neighbouring context, which is what hierarchical chunking formalises.

Common mistakes

Four mistakes recur in production chunking implementations.

The first is using the default chunk size from a tutorial without measuring its impact on the team's own corpus. The 1000-token default in many quickstart guides reflects early-RAG conventions, when corpora were small, documents were English prose, and embedding cost dominated the indexing budget. Modern embedding models, languages with longer average word lengths, and corpora dense in code or tables all change the optimal chunk size. Sweep chunk sizes between 200 and 1000 tokens against the golden-set evaluation from Issue 003 before settling on a value.

The second is omitting overlap. A chunker with no overlap places the last sentence of one chunk and the first sentence of the next in separate retrieval units, even though they often belong together semantically. An overlap of roughly ten percent of the chunk size (50 tokens for a 512-token chunk) recovers most of the boundary-loss problem at a modest storage cost.

The third is storing only the chunk text. A chunk that surfaces in the LLM's response with no metadata is impossible to attribute to a source, to rerank with positional features, or to expand to neighbouring context. Every chunk should carry, at minimum, doc_id, chunk_idx, a source page or section reference, and a stable identifier for the parent document. The cost is a few bytes per chunk; the benefit is a retrieval pipeline that can be debugged when it misbehaves.

The fourth is re-chunking on every document update without versioning. When the chunking strategy changes (a new separator hierarchy, a larger window, a different tokeniser), the embeddings produced under the old strategy and the new strategy are not directly comparable, and a mixed index degrades retrieval quality silently. Either re-chunk and re-embed every document on a strategy change, or carry a chunking_version field on every chunk so the retrieval pipeline can filter to a consistent set during a migration.

Summary

Chunking is the cheapest decision in the RAG pipeline to get right at indexing time and the most expensive to change later. Three strategy families dominate: fixed-size token chunking for uniform corpora, structure-aware chunking for heterogeneous documents, and hierarchical chunking for applications that need precise retrieval with full generation context. The right chunk size depends on the embedding model's effective context and on the document type, and the right strategy depends on what the corpus actually looks like, not on what the quickstart guide assumed. Measure the lift against the same golden-set evaluation that Issue 003 introduced, and treat the chunking decision as load-bearing for every layer downstream of it.

Production checklist

  • Profile the corpus before choosing a strategy: document heterogeneity (prose only, mixed structure, code-heavy), average and median document length, and the fraction of documents with markdown or HTML structure.

  • Start with fixed-size token chunking at 512 tokens with 50-token overlap if the corpus is homogeneous, or with RecursiveCharacterTextSplitter and its default separator hierarchy if it is not.

  • Use the embedding model's own tokeniser for chunk-size accounting, not character or word counts; otherwise chunks silently overflow the model's effective context.

  • Carry doc_id, chunk_idx, a source page or section reference, and a chunking_version field on every chunk record.

  • Sweep chunk sizes between 200 and 1000 tokens against the golden-set evaluation from Issue 003 before committing to a value.

  • Consider hierarchical chunking when retrieval precision is the binding constraint and the LLM has a generous context budget. The pattern doubles storage and adds a join at query time.

  • Re-chunk and re-embed the entire corpus on a strategy change, or filter retrieval by chunking_version until the migration is complete.

  • Re-evaluate the chunking decision annually. Embedding models improve, and the optimal chunk size shifts with them.

Further reading