A similarity threshold borrowed from the wrong comparison

Same metric, different scales

Cosine similarity gives you a figure between zero and one, and it's really easy to believe that figure means the same thing wherever you read it. It doesn't. The similarity between two long passages and the similarity between a short question and a long passage sit on different scales, and a threshold calibrated on one will wreck the other without telling you.

I hit this building a retrieval layer over a private document corpus. Retrieval returned nothing. I don't mean poor results, I mean nothing, for questions I knew the corpus answered directly, with the matching passage sitting in the table in front of me.

Where 0.5 came from

The threshold was 0.5, and the reasoning behind it sounded fine. Cosine similarity above 0.5 means two things are related, and below it they aren't. That intuition isn't made up either. It comes from comparing documents with each other, where it holds up well.

I could check it, because the same embeddings were already being used to compare chunks against each other. Two passages on the same topic from the same corpus scored between 0.74 and 0.77. Against that baseline 0.5 looks generous, since it lets in things that are only loosely related.

Then I measured the comparison the system actually performs. A short user question against those same long chunks landed at roughly 0.3 to 0.5, including for the chunk that was unambiguously the right answer. So the threshold wasn't filtering weak matches at all. It was sitting above the ceiling of the entire measurement.

Why the scales differ

The asymmetry is structural, so it isn't a quirk of one corpus.

A chunk is packed to a target token count, with an overlap carried over from the previous chunk on purpose, so that a thought split across a boundary survives in both.

export function chunk(text: string, opts: ChunkOptions): ChunkInput[]
// opts: { targetTokens, overlapTokens }

So a chunk is long by construction, and its 1536-dimension embedding is a blend of everything in it, including the main point, the digression either side of it and the sentence fragments inherited from the overlap. A user's question is a dozen words about one thing, and its embedding is concentrated.

Two long passages are two blends, and blends of related material point in broadly similar directions, which is where 0.74 to 0.77 comes from. A concentrated vector compared against a diluted one can't get there, because most of what the chunk's vector encodes is material the question never mentioned. The dilution puts a ceiling on the score that has nothing to do with relevance.

So a document-to-document threshold was just the wrong one to use for question-to-document retrieval.

Recalibrating

The fix is one number (0.5 down to 0.3), in the retrieval module's default and in the Postgres function behind it.

CREATE OR REPLACE FUNCTION public.match_chunks(
  query_embedding vector(1536),
  similarity_threshold float DEFAULT 0.5,
  match_count int DEFAULT 8
)

I didn't just pick it because "0.3 felt safer". I measured the distribution the system will actually see, found where known-good matches fall, and set the cut below that. On this corpus strong matches sat around and above 0.3, so I think anything that clears it is worth handing to the model, and the top-k limit does the real work of deciding what gets used.

That changes what the threshold is for. With a top-k of eight and ordering by similarity descending, ranking already decides which chunks are best, so picking good chunks isn't the threshold's job. It's there for one case. When nothing in the corpus is relevant at all, it should return an empty set, not eight bad chunks passed off as an answer.

Set at 0.5, it had been answering that question with "nothing is ever relevant".

When retrieval does return something, those chunks go to the model alongside the conversation history, truncated to a 30,000-token budget. The retrieved passages are the whole point of the request, and a long conversation should never be able to crowd them out.

When nothing matches

That only matters if empty is handled properly, and the natural way to handle it is wrong. Pass zero chunks and the user's question to a model and it'll answer anyway, fluently, from its own weights, which is what retrieval is there to stop.

So the chat endpoint checks before it calls anything.

if (retrieved.length === 0) {
  // stream a fixed message, persist it, close the stream
  // — the model is never invoked
}

There's no model call, no prompt and no temperature. It's a fixed sentence saying there's nothing in the corpus on that subject and suggesting the user rephrase. It's streamed in small pieces so it arrives like any other reply, and stored as an ordinary assistant message so the transcript shows what the system said.

I didn't do this with a system prompt telling the model to decline when it has no sources, because models don't always follow instructions. There's no path from zero chunks to a generated answer, because the code that would generate one never executes.

That guard is also why the threshold bug was survivable. Every question in that window returned the fallback. That's wrong, but visibly and consistently wrong, and nobody got a confident fabrication. If the empty case had fallen through to the model, the same bug would have produced plausible answers grounded in nothing, and I'd have been debugging tone, not a number.

The retrieval module carries eight tests, and I think the useful ones are the ones that check the contract. They check that the threshold and count reach the database, that an empty result comes back as an empty array, and that results arrive ordered. A test can't tell you 0.5 is the wrong constant, because 0.5 is a perfectly valid float. You only find that out by measuring.

So now, before I trust any threshold, timeout or confidence cut-off, I measure the comparison it'll actually sit on, look at where known-good cases fall, and put the line below them. Here that took me an afternoon.