Building citation-aware AI for complex documents

Citations aren’t a post-processing step

The naive approach to citations is: generate an answer, then search for which source passages look similar to what the model said. This produces plausible-looking citations that are frequently wrong — the model paraphrases, combines multiple sources, or hallucinates a claim that happens to resemble something in the corpus.

Real citation-aware AI requires the architecture to flow evidence forward: the system retrieves specific passages, assembles them as context, and the model generates answers constrained to that context. Citations point to what was actually provided as input, not what looks similar after the fact.

Chunk boundaries determine citation granularity

If your chunks are too large (full pages, entire sections), citations are vague — “see page 14” doesn’t help a lawyer verify a specific claim. If chunks are too small (individual sentences), context fragmentation makes retrieval noisy and the model loses coherence.

We’ve found that paragraph-level chunks with overlap, augmented by parent-document context injection at generation time, hit the right balance. The chunk is the citable unit. The parent context ensures the model understands how that chunk fits into the broader document. This produces citations that point to a specific paragraph while generating answers that reflect broader document understanding.

Metadata makes citations usable

A citation that says “chunk_4827” is useless. Users need document title, section heading, page number, and ideally a highlighted passage. This means the ingestion pipeline must preserve structural metadata through every stage — from parsing through chunking through indexing.

We store a metadata envelope alongside each chunk: source document ID, page range, section path (e.g., “Contract > Section 4.2 > Termination Clause”), and character offsets into the original document. At generation time, the model’s output includes chunk references that resolve directly to navigable locations in the source.

Cross-document citation creates new challenges

Legal and compliance workflows frequently need answers that synthesize across multiple documents. “What are the termination clauses across these five contracts?” requires retrieving relevant passages from each document, presenting them as separate evidence streams, and generating a synthesis that cites each source independently.

The context assembly strategy matters here. We use source-grouped context: retrieved chunks are grouped by document before being passed to the model, with explicit document boundaries in the prompt. This produces answers that naturally separate claims by source rather than blending everything into an undifferentiated summary.

Evaluation for citation quality

Standard RAG evaluation (answer correctness, faithfulness) isn’t sufficient for citation-aware systems. We evaluate three additional dimensions:

• Attribution precision: Does each citation actually support the claim it’s attached to?
• Attribution recall: Are there claims in the answer that should be cited but aren’t?
• Navigability: Can a user follow the citation to the exact relevant passage?

We generate evaluation sets by having domain experts annotate expected citations for known queries. This catches regression early — particularly after embedding model changes or chunking strategy updates that can silently break citation quality while answer fluency remains unchanged.