My Vector Search Had No Vectors

The metric was fine. The corpus underneath it was noise, and every test was green.

Three different questions. Three unrelated notes that should answer them. My retrieval returned the same seven notes for all three, in the same order, every time.

A backup checklist and a hybrid-retrieval note are not the same answer to three different questions.

I had spent the previous weeks making sure my measurement of retrieval was honest — repairing a denominator, checking the metric's ceiling, pinning down what invalidates what. All of that was real work and I would do it again. It also could not have found this, because the metric was working correctly. It was faithfully reporting the quality of a retrieval channel that was returning noise.

What I measured first

I did not start from the code. I started from a number I could not explain, and I want to be precise about the order because it is the transferable part.

One deterministic test row kept failing in a way that made no sense. The row asserts a phrase present verbatim in chunk 0 of the note the query names. Retrieval returned chunks 1 and 2 instead. I had been hunting for a ranking rule — some title-versus-subchunk preference I had gotten wrong.

So I asked the corpus a question that has a known answer: how similar are two paragraphs of the same note about the same subject?

A real text embedder puts two paragraphs of one note far above zero. A corpus whose maximum similarity between any two chunks is 0.084 is a corpus of near-orthogonal vectors. They carry no meaning at all.

That is a known-input test, same as the oracle check in the last post — except pointed one layer down. The oracle check asks can my metric respond? This asks is the thing my metric is measuring real? I had done the first and assumed the second.

There was no ranking rule to find. With near-orthogonal vectors, which chunk comes back is arbitrary. I had been looking for the wrong kind of bug for days.

The mechanism

Only after the measurement did I go read the code, and the mechanism agreed with the number.

To keep evaluation runs reproducible, I had built a decorator that answers a document embedding from a hash of its own input instead of calling the provider. That is a sound idea in one specific place: when an agent writes a conversation record mid-run, its text is freshly sampled model output, so calling a real embedder mints a cache key that will never recur. Deriving the vector from the input bytes breaks that loop, and the vector has no reader anyway — those records are excluded from retrieval by an earlier decision.

The decorator dispatched on the kind of the embedding request:

if req.Kind != llm.EmbedKindDocument {
    return e.real.Embed(ctx, req)
}

Notebook ingestion embeds every chunk with Kind: EmbedKindDocument. So the user's actual notes — a first-class retrieval route, read by the hybrid ranker on every single query — took the derived branch, exactly as a run-written record did.

Here is the part I find genuinely instructive. The package's own docstring had already named this as the one thing that must never happen:

That costs nothing exactly where this decorator is used and nowhere else […] Using this anywhere a vector IS read would silently destroy retrieval quality, which is why it is a composition-root decision named in one place rather than a flag on a request.

The reasoning is correct. The scope is wrong. I had written down the exact failure mode, identified it as catastrophic, explained my mitigation — and the mitigation did not cover the case. A comment describing a boundary is not the same as a boundary, and mine had been sitting there being right and ineffective for weeks.

Why nothing caught it

This is the question worth sitting with, because the answer is not "I needed more tests."

The tests were green because it is a quality defect, not a correctness defect. Every embedding call returned a well-formed vector of the right dimension. Nothing threw, nothing timed out, no dimension mismatched. Cosine similarity computed fine. Retrieval returned a ranked list of the requested length. Every assertion about shape passed, because the shape was perfect.

The English rows partly masked it. My retrieval is hybrid: vector plus lexical, fused. For English queries the lexical channel works, so the rankings carried real signal and looked reasonable. What was inert was the vector half. Which means the queries where this was total — CJK, where the lexical channel scores zero and vector is all there is — were the minority of my test set, and their failures read as "CJK is hard."

And the failure was invisible to the metric by construction. My candidate pool is built by running real retrieval and then having a human judge what it surfaced. So the pool was selected by an arbitrary ordering, and then I judged that arbitrary set carefully and honestly. Look at what that produced:

That last CJK query — zero relevant items out of nineteen judged — was published in my own benchmark document as a genuine retrieval miss, in the section that exists specifically to report things the system honestly failed to find. The honest-reporting habit worked exactly as designed. It reported the failure. It attributed it to the wrong cause: nothing relevant was surfaced not because the corpus lacked the answer, but because the channel that should have surfaced it was dead.

An evaluation whose reference set is built from its own system's output cannot detect a uniform degradation in that system. It will measure, faithfully and repeatably, how well the system ranks the things a broken version of itself decided to show.

The repair, and the direction that matters

The fix was to make the decorator opt in per call site rather than dispatch on request kind. One caller — the episodic write, whose input really is freshly sampled text — marks itself explicitly. Every unmarked call reaches the real embedder.

The direction is the point. Under the old design, a new ingestion path silently got derived vectors and destroyed its own retrieval quality. Under the new one, a new path is semantic by default and has to ask for anything else. Deriving where a vector is read fails silently, so the default has to be the safe side.

Then the vectors, measured the same way:

Two paragraphs of one note went from negatively correlated to 0.87.

And the deterministic row I had originally been chasing? It passes. Against its original assertion, unmodified. The bug I had been hunting never existed.

What the repair cost, and what it does not prove

The pool had to be rebuilt, because it had been selected by broken retrieval. The existing human judgments survived — a judgment is a call on a (query, candidate) pair, so a rebuilt pool that surfaces the same candidate keeps its judgment. What a rebuild changes is which candidates appear: it adds ones the broken channel never surfaced, and those arrive unjudged. So the cost was a labelling session bounded by how much better retrieval got, not a rebuild of the label set.

I want to be careful about the numbers on the other side of this, because the temptation is to present a before-and-after and call it improvement. I can't. Between the old published figures and the current ones, the denominator changed, the judgment set was corrected and expanded, episodic fixture dates moved, the query set grew from 24 to 28, and the lexical channel changed. The current numbers are a new baseline, not a correction of the old ones toward their true values. The old ones measured something else and are retired rather than converted. A delta across them would be a number with no meaning, and my benchmark document says so in its own voice, directly above the figures.

What I can state is the mechanism: the notebook vector channel produced meaningless vectors, and now it produces meaningful ones, measured by a known-input test before and after.

The generalisation

A known-input test on your data is as necessary as one on your metric. Two paragraphs of one document should be similar. If they are not, stop. You already know the answer to that question, which is exactly what makes it a usable instrument — same shape as feeding a metric a perfect ranking and seeing whether it reports 100%.

Quality defects do not throw. An embedding pipeline that returns well-formed garbage passes every structural assertion you have. If your tests only check shape, they are checking the thing least likely to be wrong.

A comment naming a failure mode is not a guard against it. I had written the failure down, called it catastrophic, and described a mitigation that did not cover the case. Prose does not execute. If a boundary matters, something has to enforce it, and the enforcement should fail in the safe direction — opt in to the dangerous behaviour, never out of it.

An evaluation built from your system's own output cannot see a uniform degradation of that system. Pooling is still the right way to build a judged retrieval set — I would not give it up. But it inherits your system's blind spots wholesale, and no amount of careful human judging repairs that, because the judging happens downstream of the selection. The check has to come from outside: a known input, a property you can assert without consulting the system.

---

One thread from a longer set of engineering notes on building taidle — a knowledge base with an LLM agent over it, where every reply renders the route the system actually took. The essays and the 16 architecture decision records are at github.com/matthewhoung/taidle-labs.