RAG Chunking Is the Product (The Model Is Just the Engine)
Your RAG demo worked on ten pristine PDFs. Then legal uploaded a 400-page policy with footnotes, exhibits, and cross-references that the chunker turned into word salad.
Your RAG demo worked on ten pristine PDFs. Then legal uploaded a 400-page policy with footnotes, exhibits, and cross-references that the chunker turned into word salad. The model didn't fail — your scissors did.
Here's the thing: chunking is where domain knowledge enters. Structure-aware splits (headings, clauses, tables), overlap tuned to citation needs, metadata that survives retrieval — that's the product. Swap embedding models all day; if chunks splice mid-sentence across section 4.2, users get confident nonsense.
Chunk like a librarian, not a shredder
Respect document structure. Keep tables intact or summarize them explicitly. Propagate titles into chunk headers so embeddings carry context. Test retrieval with real questions — not "what is this doc about" but "can we expense client dinners in Q3 per policy."
Can a human answer from this chunk alone? If not, neither can the model.
- Semantic boundaries — headings, numbered clauses, API endpoint blocks
- Overlap with purpose — bridge context, don't duplicate whole sections
- Version tags — stale chunks are worse than no chunks
- Human spot checks — sample 50 chunks before you trust the index
RAG is a publishing problem wearing an AI costume. Editorial discipline beats bigger embeddings.
| Fixed tokens | Structure-aware | |
|---|---|---|
| Policies | Split mid-clause | Clause-complete |
| Tables | Fragmented | Preserved or summarized |
| Citations | Vague | Section-linked |
Tables, exhibits, and the footnote trap
Legal and finance docs don't chunk like blog posts. A table split across two embeddings is worse than no table — the model will invent relationships between orphan rows. Exhibits referenced from page 200 need parent metadata, not isolated shards. Footnotes that redefine terms mid-document will poison retrieval if you treat them as body text.
Here's the thing: invest in a document parser that understands your genres — contracts, policies, API references, SAP config guides — not a generic splitter. Parser quality is chunk quality.
- Table strategy — keep intact, or generate natural-language summaries per table
- Cross-references — store "see Section 4.2" as links in metadata, not orphaned text
- Effective dates — version chunks; retrieval must filter superseded policy
- Language tags — multilingual corpora need chunk-level locale, not doc-level guesses
Your chunker is an editor. Bad editing makes PhD models look like interns.
Eval questions that actually hurt
Skip "summarize this document." Use adversarial retrieval questions: clauses that contradict each other, exceptions buried in appendices, numbers that only appear in tables. If recall@5 fails on those, no reranker saves you — fix the scissors.
What this means for you: sample fifty chunks manually before indexing ten thousand docs. If a human can't answer from the chunk alone, rewrite the split logic. Run chunk versioning in CI when parsers change — re-embed is expensive; catching bad splits early is cheap.
"What section is this from, and what question would this chunk answer?" — if you can't tell, neither can the model.
Pipeline ownership
Assign a named owner for the ingestion pipeline — not "the AI team." When legal updates a policy PDF, who re-chunks and re-embeds? If nobody owns it, your index rots while the chat model gets upgraded quarterly.
What this means for you: monitor index age per document source. Alert when critical corpora pass SLA without re-index. RAG quality decays silently; dashboards should scream before users do.
Multilingual chunking
Mixed-language corpora need chunk-level language detection — not document-level guesses. Embed with language-appropriate models or unified multilingual encoders tested on your acronyms in every locale you operate.
What this means for you: eval retrieval per language separately. A chunker that works in English and fails in German kills global rollouts quietly.
Fix chunking before you fix the chat model — the narrator can't recover facts retrieval never found.
What this means for you: give chunking pipeline its own on-call rotation when RAG is production-critical.
Fix the scissors before you blame the model
That 400-page policy didn't break because the embedding model was weak — it broke because the chunker spliced mid-sentence across section 4.2 and handed the model confident nonsense to work with. What this means for you: sample fifty chunks by hand before you trust an index of ten thousand documents. If a human can't answer from one chunk alone, neither can the model, no matter which reranker you bolt on.
Here's the thing: name an owner for the ingestion pipeline the same way you'd name one for a production service. RAG quality decays silently when policies update and nobody re-chunks — the chat model gets praised for stability while the index quietly rots underneath it.
Your chunker is an editor. Bad editing makes PhD models look like interns.
Your move: Take your five worst RAG failures from production — the queries that returned irrelevant chunks or missed critical context. For each one, trace the failure to a chunking decision: Was the boundary in the wrong place? Was the chunk too large or too small? Did cross-reference context get lost? Document the pattern, then fix your chunking strategy for that document type. Repeat until production failures drop below 5%.
Stay with us · decision
How Do You Handle Document Chunking in Your AI Projects?
When implementing RAG systems, how do you approach document chunking to ensure accurate and meaningful retrieval?
No account needed — pick a take, then keep reading. We rotate these prompts so each piece feels like a conversation, not a clone.
