Exhibit data not captured yet.
node scripts/rag/capture-exhibit-miranda-nist.js
A high score is not proof of a right answer. This page shows why, using two real documents and the real retrieval engine behind this site.
How a Document Comes In Shapes What Comes Out
Miranda arrived as plain text. NIST arrived as a structured PDF. Same chunker, same pipeline — but only one of them started with information the system could use to find its own structure.
Inspect how a document enters your system before trusting what comes back out.
A chunk is a piece of a document, cut down to whatever size an embedding model can hold in its head at once — a few hundred words, occasionally a bit more. This system does not read your document. It was never going to. It reads the chunks, compares them to your question, and hands the model whichever pieces scored closest. If a chunk got cut badly — mid-argument, mid-table, mid-sentence — the piece that actually answered your question may not exist anymore, not as a single retrievable thing. Nobody tells you when this happens. The system does not know it either. Below, you can watch it happen to two real documents, on purpose, so you don’t have to find out the hard way on one that matters to you.
The full pipeline is on the how this works page.
Two documents, one chunker
The chunker is deterministic: same document, same parameters, same output every time. What changes between Miranda and NIST is not the chunker — it is what the source document gave the chunker to work with.
Miranda came in as plain text — the extraction path never captured any visual layout information, because there wasn’t a PDF to read it from. NIST came in as a real PDF, carrying its own font sizes and page layout. That difference in how each document entered the pipeline — not any difference in how well either one was written, scanned, or preserved — is what determined whether the chunker could find internal structure, and whether retrieval could later tell you which section an answer actually came from.
The Harlan query
Let’s make this real. Imagine you ask a legal assistant: “What did the one judge who disagreed actually argue?” You want his words — not the majority’s, not anyone else’s. Below, we asked our system exactly that question about Justice Harlan’s dissent in Miranda — once with an older, careless version of the system, and once with a fixed one.
Notice both numbers moved the same direction. The score went up, and the answer became correctly labeled. That’s what it looks like when retrieval works: relevance and correctness point the same way.
The old version couldn’t do that. It didn’t know this document had a dissent, so it had no way to go looking for one — it just handed back whatever chunk scored highest, with total confidence, no label, no way for you to check it. In this case, that chunk runs two justices together with no line between them: the tail end of a different justice’s dissent, flowing straight into the real opening of Harlan’s. We asked the system this exact question against that chunk. It answered confidently, and got real Harlan material right — but it also lifted a line from the other justice’s dissent and presented it as part of Harlan’s argument, cited exactly like everything else. Nothing about the answer would have told you which line was whose.
A high score does not mean a right answer. Knowing where an answer came from is what makes it trustworthy.
An honest complication
Not every fix wins cleanly, and we’re not going to pretend this one did. Here’s a case where extracting the table made the document easier to read — and still lost, on the numbers, to the messy version. Worth sitting with.
The NIST table story is subtler. The query below targets the GOVERN function’s categories and subcategories — content that lives in a structured table in the PDF.
Comparison axis: one chunk, the GOVERN table region — its score before extraction (v1, garbled column text) and after (v2, clean extracted table).
In v1, the GOVERN table region was split into size-based chunks, with column text interleaved across rows — a classic two-column PDF extraction artifact. In v2, the table was extracted as a single clean chunk. The v1 garbled chunk ranked higher for this query than the v2 clean extracted table. Cosine rewards keyword repetition from the column interleaving; it cannot distinguish readable structure from the garble. The extraction improved the corpus’s comprehensibility — not its retrieval score for this query.
The Miranda cosine dip (from the discovery data) reflects a chunk-size effect from heading-recovery re-phasing; the next exhibit examines it.