The integration worked beautifully
In my hybrid-search lab, a local LLM generated query variations for a keyword retrieval system. The pipeline combined BM25, reciprocal rank fusion, spelling correction, and constraints such as required phrases or exclusions. The goal was to handle the intent behind a shopper’s words without building an embedding index for the catalog.
At the integration level, the local model looked dependable. The recorded llama3.1:8b Q4 runs on an RTX 5050 returned valid JSON across 619 calls, with no recorded errors. Warm calls were roughly half a second in that group of runs.
That is a useful engineering result. It says the model could repeatedly produce the shape of data the application expected. It does not tell us whether the additional terms improved retrieval. For that, I needed a different measurement.
The baseline that did less
The lab used human relevance labels from Amazon’s ESCI dataset, with a 39,000-product sample and a broader evaluation set of 300 queries. The comparisons included keyword search without expansion. That simple baseline matters: a new stage has to justify disturbing results that already work.
A model can generate plausible related words that broaden a precise query in the wrong direction. Reciprocal rank fusion can combine ranked lists, but it cannot guarantee that the additional lists express the shopper’s actual intention.
This led to a gate: use the model when the keyword evidence is weak, and preserve the fast path when it is strong. Deterministic spelling correction against the catalog vocabulary also belonged before a model call. Some failures needed a better token, not a generated interpretation.
The number that survived a rerun
The notes contain an important correction. An early reading suggested a small gain from the API-model comparison. But the two sides were sampled at different temperature settings, and later measurements did not support that conclusion.
At matched temperature, a 100-query comparison ran each configuration twice to inspect its own variability. Local 8B expansion reduced precision at ten by 0.0195 against keyword-only retrieval; its redraw spread was 0.0010. The API-model arm moved by −0.0030 with a redraw spread of 0.0080.
Precision at ten asks how many of the first ten results meet the relevance criterion. The local model’s negative movement was much larger than its observed redraw spread. The API arm’s movement sat inside its spread, so the earlier positive claim was withdrawn.
This still did not prove one model was better than the other. The gate examined the stability of each arm; it was not a paired statistical comparison between them. It also did not establish how either model would behave on a fresh set of weak-keyword, conversational queries.
| Matched-temperature arm | Δ P@10 vs keyword | Redraw spread |
|---|---|---|
| Local llama3.1:8b | −0.0195 | 0.0010 |
| API comparison | −0.0030 | 0.0080 |
The evaluator can have a blind spot too
Later reranking experiments produced another useful warning. Many retrieved candidates had no relevance label. Under the evaluation convention, those unjudged items counted as irrelevant, even when a spot-check found a sensible product.
An oracle that knows which items carry positive labels can exploit that information in a way a real ranker cannot. The notes explicitly warn against treating its apparent headroom as an achievable target. Optimizing toward the label coverage could look like improving the search engine.
That does not make the benchmark useless. It tells me to interpret it with human spot-checks and judged-only analysis, and to keep the dataset, language, sample size, and scoring convention attached to every claim.
What changed in my thinking
I now treat format reliability, model capability, and product usefulness as separate checkpoints. A schema validator establishes the first. A well-designed task evaluation gets closer to the second. The downstream result is where the third has to be earned.
The work also reinforced why I keep rejected conclusions in the record. If the old small positive number survives while the correction disappears, the next person can make an entirely reasonable decision from the wrong evidence.
Sometimes the strongest use of an LLM is knowing which request should reach it. Sometimes the most valuable research result is the tempting claim you no longer make.