Skip to article
david sheffer.
← All notes

Local LLMs · A failed experiment

I made it stream. It got slower.

A 70.5-second local-model pipeline became a 109-second pipeline after adding progressive output. The missing ingredient was actual overlap.

Two models, one answer, a long wait

One part of my local chat research paired Qwen3-30B-A3B with DictaLM for Hebrew output. Qwen produced an English answer, then DictaLM translated it. For a long response in the August 6 notes, the single-call translation path took 70.5 seconds end to end.

That is a long time to stare at an interface. Progressive output seemed like a practical next step: take chunks from the first model, translate them, and show them as they become available. Smaller pieces, earlier feedback, perhaps a faster finish.

The measurements did not follow that story. Fixed batches of 320 characters took 109 seconds. Translating one sentence at a time took 96 seconds and introduced additional quality problems: an untranslated sentence escaped, and numbered lists broke apart.

Recorded designTotal time
One translation call70.5 seconds
320-character batches109 seconds
One call per sentence96 seconds
One documented long-answer experiment, August 6, 2026. These timings describe the tested pipeline and response, not a distribution across all prompts.

The loop was still serial

The control flow explained much of the surprise. The generator read output from Qwen, blocked while a translation call completed, and then returned to reading. Splitting the answer into chunks did not create a producer that kept draining the first model while the second worked.

Each new translation call also brought another prompt-processing cost. We had added repeated setup work without establishing the scheduling overlap we hoped would pay for it. The interface looked more active while the overall job took longer.

A response can be streamed to the browser while its backend remains sequential. That is an easy detail to miss when the user-visible behavior is your only instrument. The timing record forced me to inspect the actual execution path.

Even real overlap needs a test

The next hypothesis was a producer that continuously reads Qwen’s stream while translation consumes completed chunks. If scheduling is the bottleneck, overlapping two stages can move completion time toward the slower stage instead of their sum.

But both models were running on the same memory system. The decode measurements already pointed to bandwidth pressure. Concurrent generation could make the two models compete for the same constrained resource. Moving work onto another thread would not create a second memory bus.

The notes therefore proposed a cheap test before another rewrite: measure each endpoint alone, then both together, and compare aggregate throughput. The suggested speedup was a prediction. The document did not establish that the new pipeline achieved it.

Earlier text is a different metric from a finished answer

This experiment also made me separate time to first useful output from time to completion. Progressive output can improve the feeling of responsiveness even when it does not make the entire answer faster. Both can matter; they need separate measurements.

“Useful” matters too. An early fragment that breaks the list structure or exposes text in the wrong language is not automatically an improvement. Chunk boundaries affect translation context, formatting, and the reader’s ability to follow the answer.

My next evaluation would keep the prompt and output budget fixed, measure the first complete useful unit and the final answer, and review language and formatting alongside latency. Without those controls, a shorter response can masquerade as a faster system.

Why I kept the failed result

The useful artifact was the failure record: the exact batching design, its time, the blocked read loop, and the remaining bandwidth hypothesis. That record prevents a future session from presenting the same rewrite as an obvious optimization.

This is where my work on local models meets my work on engineering memory. A failed experiment becomes valuable context when it preserves the reason it failed and the question it did not answer.

The next idea can still be better. It just has to start from the system we measured.

NEXT NOTEPerfect JSON. Worse search.