Your Chunking Strategy Is Clever. Your Citations Still Point at the Wrong Paragraph.
Your retrieval team spent a month arguing about recursive character splitters versus late chunking. The demo deck has a neat diagram. Compliance is happy because every answer shows a citation. Then a solicitor opens the cited paragraph and finds a neighbouring clause that does not support the claim. The model was confident. The pointer was polite. The evidence was wrong.
That gap is becoming the quiet failure mode of enterprise RAG. Chunking got clever. Citation UX got pretty. Alignment between the span the model used and the span the user sees did not keep up.
Picture a policy assistant for an Australian bank. An adviser asks whether a customer can roll a product under a grandfathered fee schedule. The answer cites Policy Manual v4.2, section 7.3. The highlighted box shows the introduction to section 7. The decisive exception lives three paragraphs later in the same chunk, or worse, in the next chunk that never made the top three.
Or a clinical guideline bot. Nursing staff get a tidy answer with a PDF page number. The page has two columns and a callout box. Your chunker sliced by token count across the gutter. The citation lands on the left column. The dosage note the model paraphrased sat in the callout on the right.
Or a contracts assistant. Legal ops ask about termination for convenience. The model quotes language that looks right. The citation opens the definitions section because your overlapping windows put the defined term and the operative clause in different retrieval hits, then the answer assembler attributed the wrong one.
None of these teams are careless. They shipped embeddings, hybrid search, and a citation panel. They still cannot prove that the highlighted span is the span that grounded the sentence.
Modern chunking optimises for retrieval quality. Semantic boundaries, parent document retrieval, small-to-big, hierarchical indexes. Those tactics improve recall. They also loosen the contract between retrieval unit and display unit.
When you retrieve a parent and generate from children, the UI often cites the parent. When you retrieve a child and expand context for the prompt, the UI often cites the child while the model leaned on the expansion. When you merge overlapping windows, the winning chunk id may not be the window that contained the decisive sentence.
Rerankers make it murkier. They reshuffle passages after the first pass. If your citation logic still points at the first-stage hit ids, users open a paragraph the model barely used. If you cite the reranked top hit but the prompt concatenated three passages, the single citation is theatre.
Multimodal and OCR pipelines add more drift. Page numbers from the PDF viewer do not match layout-aware chunk offsets. Tables get serialised into text that no longer sits where the screenshot suggests. Your citation says page 14. The grounding sat in a reconstructed table that never existed as a continuous paragraph.
Wrong citations are not just an IR metric miss. They are a governance miss.
Risk and legal treat citations as an audit trail. If the trail points at the wrong paragraph, you have manufactured false confidence. A reviewer who spot-checks citations will either rubber-stamp nonsense or lose faith in the whole system. Both outcomes are expensive.
Customer-facing teams learn the wrong lesson. They stop opening sources because the panel feels authoritative. When something goes wrong, nobody can reconstruct which span actually entered the context window. You get screenshots of the UI and logs of chunk ids that disagree.
Regulators and internal audit will not accept "the model probably used nearby text". Nearby is not evidence.
Treat citation as a first-class product contract, not a decoration on the answer card.
Store provenance that survives every transform. For each token span that enters the model context, keep the source document id, version, byte or character offsets, page or section anchors if available, and the chunk id used for retrieval. When you expand parents or merge overlaps, retain the leaf spans that actually went into the prompt.
Cite what was used, not what was retrieved. If three passages entered the context, expose three citations or a ranked set with clear primary and supporting labels. Do not collapse to a single pretty link because the design system prefers one chip.
Highlight the exact grounding span in the source viewer. If you cannot resolve offsets into a stable highlight, do not pretend. Show the document and section with a warning that precise span mapping failed. Honesty beats a confident wrong box.
Evaluate citation faithfulness as hard as answer quality. Sample production queries. Ask reviewers whether the cited span supports the claim. Track "answer right, citation wrong" as its own failure class. It is common, and it is invisible if you only score the prose.
Keep chunking experiments on a leash. Every new splitter or hierarchy must declare how citations will resolve. If the strategy cannot map generated claims back to displayable spans, it is not ready for regulated workflows.
If any of those fail, your chunking strategy can still be clever. Your citations are still pointing at the wrong paragraph.
Enterprise RAG earns trust when evidence is inspectable. Clever chunking without faithful pointers is a demo costume. Wire provenance through retrieval, prompting, and the source viewer, then measure citation support like you measure accuracy. Until the highlighted paragraph is the one that grounded the sentence, you do not have a citation. You have a gesture.
