The painful truth about chunking

Most RAG systems miss answers they already have. Not because the embedder is bad, but because the content was chunked in a way the model cannot retrieve. I see the same pattern in audits: 1k token chunks with 200 token overlap, a vector DB set to k=5, and hope. Then teams add a re-ranker, crank up k, and wonder why latency and cost explode while recall barely moves.

Where chunking fails and why

This problem shows up when:
– Docs are semi-structured: API references, policy manuals, RFCs, financial PDFs
– The answer lives across boundaries: a definition in one section, constraints two paragraphs later
– You have tables, code, or lists that got split into nonsense during parsing

Why it happens in real systems:
– Fixed-size chunking ignores document structure, so relevant context is either diluted or cut in half
– Overlap creates duplicate near-identical chunks that confuse re-rankers and waste tokens
– PDF parsing loses layout, so row headers and footnotes drift away from the values they qualify
– Teams tune k and prompts but never measure retrieval quality with a real eval set

What most teams misunderstand:
– Bigger chunks are not safer. You reduce miss-at-boundary errors but inject multiple topics per chunk, which kills precision
– Overlap is not a fix. Overlap mainly increases index size and latency. It only helps when your boundary detection is weak
– Re-rankers do not rescue missing content. If recall@50 is low, cross-encoders just shuffle the wrong candidates

Technical deep dive: architecture, trade-offs, failure modes

Think in layers, not knobs:
– Ingestion: parse, normalize, segment into semantic units, capture structure
– Indexing: multiple granularities, metadata, adjacency
– Retrieval: hybrid lexical + dense, adjacency expansion, re-ranking
– Assembly: stitch neighbors, dedupe, budget tokens to the generator

Key trade-offs:
– Chunk size vs precision: smaller chunks improve precision; larger chunks improve recall only if structure is respected
– Overlap vs index load: more overlap increases total tokens indexed by roughly size/stride; at 20% overlap the overhead is about 25% regardless of base size
– Multi-scale indexing vs cost: storing sentence and paragraph embeddings raises index size, but boosts recall for compositional queries

Common failure modes I keep seeing:
– Fragmentation: tables, code blocks, and lists split mid-entity
– Topic mixing: headings and subtopics merged into one giant chunk
– Semantic duplication: heavy overlap creates near-duplicates that dominate the top-k and drown out diverse candidates
– Layout loss: PDF lines reflowed without preserving column or header relationships
– Drifted context: the title or section label is not carried into the chunk, so queries that rely on taxonomy miss

Practical chunking strategies that work

Here is what has consistently improved retrieval quality in production.

1) Structure-aware chunking as the default

• Use document structure as the primary boundary: H1/H2/H3 for markdown, section titles for docs, function or class for code, row or logical block for tables
• Prepend a compact header to each chunk content at index time: doc_title > section_title > subsection_title. Keep it short. Do not paste the full path into every chunk
• Store the header both as text and as a separate embedding vector. At retrieval time, fuse content score and header score with a learned or tuned weight

Why it helps: this increases precision without inflating chunk size. It also improves queries that use taxonomy language like billing limits or retry policy that may appear in titles but not in the body.

2) Multi-scale indexing with adjacency

• Index two granularities for prose: paragraph chunks (120 to 300 tokens) and sentence chunks (~1 to 3 sentences)
• Keep a stable chunk_id and neighbor pointers: prev_id, next_id, parent_section_id
• Retrieval path:
  1. Hybrid retrieve top N paragraphs with dense + BM25 (reciprocal rank fusion works fine)
  2. Expand each candidate with up to M adjacent neighbors using pointers and section boundaries
  3. Optionally re-retrieve sentences within those paragraphs to pinpoint exact claims

Why it helps: you get precision from small units and recall from adjacency expansion without bloating the index with heavy overlap.

3) Table- and code-aware chunkers

• Tables: chunk by logical row set with headers inlined once. Represent each row as json-like text with column names. Add table_title as metadata
• Code: chunk by function or class using the AST. Include the docstring and signature together. Do not split long functions by tokens unless you must; if you do, add a short continuity marker and keep a neighbor link

Why it helps: models retrieve facts, not line breaks. Preserving logical units beats token windows every time.

4) Adaptive overlap based on boundary confidence

• For clean markdown or HTML with strong structure, use 0 to 10% overlap
• For messy OCR or hard wraps, use 10 to 20% overlap and increase adjacency expansion instead of overlap beyond that
• If your parser computes sentence boundaries, align chunk edges to sentence ends. This lets you lower overlap while keeping completeness

5) Retrieval-time composition, not index-time bloat

• Do not embed the same text at 3 window sizes with large overlaps. Instead, retrieve a precise candidate, then compose the final context by stitching neighbors and the local header
• Keep context assembly strict: dedupe by hash, maintain section order, cap per-source contributions to avoid one noisy doc monopolizing context

6) Hybrid retrieval that actually complements embeddings

• Use BM25 or SPLADE alongside dense. Fusing scores increases robustness for rare terms, identifiers, and numbers
• Keep k focused. Typical working set I like: dense k=200, lexical k=200, fuse, de-dup by source, then cross-encode top 40 to 80, assemble final context of 1.5k to 3k tokens

7) Metadata that matters

• Store: doc_id, section_path, chunk_index, neighbors, mime_type, layout_hints (table, code, list), updated_at
• Use section_path filtering at query time to cut down the candidate space when the user picked a product area or version

Recommended defaults by content type

• Product docs and knowledge bases: paragraphs of 160 to 240 tokens, 10% overlap, headers prefixed, sentence index optional
• PDFs with tables: parse layout with a PDF engine that preserves coordinates, chunk rows with headers inlined, add footnotes to the row’s metadata
• Codebases: function or class chunks using AST, no overlap, links to imports and callers as metadata for expansion
• Tickets and chat logs: session windows of 200 to 300 tokens bounded by timestamps and speaker turns. Do not mix threads

How I evaluate chunking changes without guessing

You will not tune this by eye. Set up a small but honest eval loop.

• Build a query set of 150 to 500 real user questions. Label gold passages and acceptable alternates. If you cannot, start with weak labels from your current system and spot check
• Metrics that matter:
  • Passage Recall@50 and nDCG@10 at retrieval stage
  • Answer exact citation rate and answer quality with a fixed generator
  • Latency p95 at retrieval and total token usage per answer
• Protocol:
  • Fix model, prompts, and re-ranker. Only change chunking and retrieval configs
  • Run ablations: base vs structure-aware, single-scale vs multi-scale, overlap 0% vs 10% vs 20%
  • Keep the best on both retrieval and cost, not just one

A real example pattern I keep seeing: moving from 1k token naive chunks with 20% overlap to paragraph-level 200 token chunks with header prefix and adjacency expansion improves passage recall@50 by 12 to 25 points, with lower latency after you drop overlap and use smarter fusion.

Cost and performance impact in plain numbers

Index and inference cost scale with how many tokens you embed and how many vectors you search.

• Overlap tax: with chunk size S and overlap r, your token inflation is roughly S / (S − rS). At 20% overlap this is 1.25x. You pay that every embed refresh
• Smaller chunks increase vector count, which can raise search latency. You offset that with filtering by section_path, hybrid pre-filtering, and lowering overlap
• Dimensionality: 384 to 512 dims is often enough for RAG. 768 or 1024 only pays off for very long tail domains. Halving dims usually cuts index RAM and speeds up queries by 30 to 40% with minor quality loss if your chunking is good
• Re-ranker budget: cross-encoding 40 to 80 candidates is plenty when your candidate pool is diverse. If you need 200+, you likely have duplicate chunks and poor structure

A quick back-of-envelope for a 1M token corpus:
– Naive: 1k chunks with 20% overlap -> stride 800 -> 1.25M embed tokens. Fewer vectors but mixed-topic chunks hurt precision
– Structured: 200 token paragraphs, 10% overlap -> stride 180 -> 1.11M embed tokens but 5x vectors. With filtering and approximate search, p95 often drops because candidates are more relevant and you re-rank fewer duplicates

The business effect: fewer hallucinations, higher first-pass answer accuracy, and lower context tokens per response. Teams usually reclaim 20 to 40% of context tokens and cut re-ranker candidates by half once chunking is fixed.

Key takeaways

• Chunk on structure first, size second
• Keep chunks small enough to be single-topic, then recover recall with adjacency expansion
• Use hybrid retrieval and score fusion to cover rare terms
• Reduce overlap and invest in a better parser
• Index at two granularities if your domain is compositional
• Evaluate with fixed models and prompts. Change one thing at a time

If you want help

If you are sitting on a RAG system that sort of works but misses obvious answers, the chunking and retrieval stack is usually the cheapest win. If you want a fast audit or need a hands-on fix, this is exactly the kind of thing I help teams clean up before scaling traffic.