Exhibit data not captured yet.
node scripts/rag/capture-exhibit-embedding-mismatch.js
We tested whether a legal-tuned embedding model quietly fails on literary text it wasn’t built for. On this document, with these six queries, it mostly didn’t — and that null result is itself the lesson: don’t assume a domain mismatch you haven’t actually measured.
We Tested for a Domain Mismatch. Mostly, We Didn’t Find One.
War and Peace is embedded in production with voyage-law-2 —
a legal-domain model. Six honestly-chosen queries against its real embeddings,
and an honest read of what actually came back.
An untested assumption about domain fit is not evidence. Here is what happens when you actually measure it, on one document, with real queries and real embeddings.
An embedding model turns a piece of text into a list of numbers — a vector — positioned so that texts about similar things end up near each other and texts about different things end up far apart. It learns that positioning from whatever it was trained on. A model trained mostly on legal writing gets good at placing contracts, statutes, and case opinions sensibly relative to each other; it was never shown a 19th-century Russian novel while learning that. “Domain fit” is just whether the material you’re actually asking it to place resembles what it learned to place well. When the fit is bad, nothing crashes and no error appears — the model still returns a ranked list, with confident-looking numbers attached, for absolutely any text you give it. The only way to know whether fit is a real problem for your document is to run real queries against it and look at what actually comes back, which is what the rest of this page does with one legal model and one novel that has no business being in its training data.
The full pipeline is on the how this works page.
A real natural experiment, not a hypothetical
The other two exhibits in this lab show voyage-law-2 working well on
legal documents — Miranda, NIST, the Federalist Papers, and the rest of this
workbench’s curated corpus. War and Peace sits in the same production database,
chunked and embedded through the same pipeline, but it is not a legal document at
all. That's a live, standing test of domain fit: what happens when a legal-tuned
embedding model is asked to retrieve literary text it was never tuned for?
We tested this with six queries, chosen for variety before any results were seen: three lexical-match queries that share vocabulary or a named scene with the actual text, and three thematic queries that require conceptual understanding rather than keyword overlap. The split is informed by an earlier finding on the Federalist Papers, where thematic queries tended to miss the retrieval floor while lexical-match queries tended to clear it comfortably. The question here was whether War and Peace would show the same pattern.
What we actually found
The six queries, top result each
Each query below ran once against War and Peace’s real, existing chunk embeddings — no re-chunking, no re-embedding of the document itself. The top-5 results for each are captured in the underlying artifact; the top hit is shown here.