Two models, one answer, a long wait
One part of my local chat research paired Qwen3-30B-A3B with DictaLM for Hebrew output. Qwen produced an English answer, then DictaLM translated it. For a long response in the August 6 notes, the single-call translation path took 70.5 seconds end to end.
That is a long time to stare at an interface. Progressive output seemed like a practical next step: take chunks from the first model, translate them, and show them as they become available. Smaller pieces, earlier feedback, perhaps a faster finish.
The measurements did not follow that story. Fixed batches of 320 characters took 109 seconds. Translating one sentence at a time took 96 seconds and introduced additional quality problems: an untranslated sentence escaped, and numbered lists broke apart.
| Recorded design | Total time |
|---|---|
| One translation call | 70.5 seconds |
| 320-character batches | 109 seconds |
| One call per sentence | 96 seconds |
Even real overlap needs a test
The next hypothesis was a producer that continuously reads Qwen’s stream while translation consumes completed chunks. If scheduling is the bottleneck, overlapping two stages can move completion time toward the slower stage instead of their sum.
But both models were running on the same memory system. The decode measurements already pointed to bandwidth pressure. Concurrent generation could make the two models compete for the same constrained resource. Moving work onto another thread would not create a second memory bus.
The notes therefore proposed a cheap test before another rewrite: measure each endpoint alone, then both together, and compare aggregate throughput. The suggested speedup was a prediction. The document did not establish that the new pipeline achieved it.
Earlier text is a different metric from a finished answer
This experiment also made me separate time to first useful output from time to completion. Progressive output can improve the feeling of responsiveness even when it does not make the entire answer faster. Both can matter; they need separate measurements.
“Useful” matters too. An early fragment that breaks the list structure or exposes text in the wrong language is not automatically an improvement. Chunk boundaries affect translation context, formatting, and the reader’s ability to follow the answer.
My next evaluation would keep the prompt and output budget fixed, measure the first complete useful unit and the final answer, and review language and formatting alongside latency. Without those controls, a shorter response can masquerade as a faster system.
Why I kept the failed result
The useful artifact was the failure record: the exact batching design, its time, the blocked read loop, and the remaining bandwidth hypothesis. That record prevents a future session from presenting the same rewrite as an obvious optimization.
This is where my work on local models meets my work on engineering memory. A failed experiment becomes valuable context when it preserves the reason it failed and the question it did not answer.
The next idea can still be better. It just has to start from the system we measured.