An ordinary CPU. An unusually large model.
My local-model research reached a concrete milestone: Qwen3-30B-A3B running through llama.cpp on a CPU-only machine with six Zen 2 cores and 64 GB of RAM. This was a dedicated server with desktop-class hardware, not a GPU workstation. “Local” here means inference on a machine I operate; it does not mean I measured these results on a laptop.
That distinction matters, because a headline about running thirty billion parameters can hide almost everything that makes the result useful. Which model architecture? What precision? How much context? One request, or several? A model loading successfully answers only the first question in a much longer experiment.
The deployment used a Q4_K_M GGUF of roughly 18 GB. In the July 29 deployment notes, raw generation was recorded at 16 tokens per second. Later August 6 chat measurements recorded roughly 18–19 tokens per second. Those are separate observations with different runtime conditions, not a controlled before-and-after speedup.
The two numbers hiding inside “30B”
Qwen’s model card lists 30.5 billion total parameters and 3.3 billion activated parameters. It is a mixture-of-experts model: a routing mechanism selects a subset of expert networks for each token. The model card lists 128 experts, with eight activated.
This changes the arithmetic of generation. The entire collection of experts is part of the model, but each token does not perform the same work as a dense model of comparable total size. That is why comparing models by the largest number in their names can lead you in the wrong direction.
It does not turn the model into a 3.3-billion-parameter file. The other experts still have to be available. Sparse activation reduces part of the compute and weight-access work; it does not erase the storage and memory requirements of the full model.
Fitting in memory is the beginning
A rough lower-bound calculation is useful: 30.5 billion weights at four bits each is about 15.25 billion bytes. That is arithmetic, not a GGUF size prediction. Mixed-precision tensors, quantization metadata, and other data push the real file above it.
The runtime needs more than that file. Context brings a key/value cache, operations need buffers, and the operating system needs room to breathe. Concurrency adds another dimension. A machine that comfortably serves a short single request can behave very differently with a long context or competing processes.
One later experiment made that painfully clear. A second large-model instance was started while the existing service held about 43 GB with a 65,536-token context configuration. The machine swapped roughly 6 GB. A baseline test crashed, and another trial produced prose at about 1.1 tokens per second.
The research note labels that run invalid. It measured memory pressure and swapping, not the speculative-decoding technique we meant to test. Calling it a failed decoding method would have preserved the wrong lesson.
Why more threads did not make the answer arrive faster
The August notes separate prompt processing, or prefill, from token-by-token decoding. Increasing the thread setting from eight to twelve improved prefill by 22% in that experiment. Decode speed stayed effectively unchanged.
The measurements pointed to memory bandwidth as the limiting factor during decode. More workers cannot make data cross the memory bus indefinitely faster. Prefill and decode are different workloads, so one setting can help the first phase without helping the second.
This changed the question I ask before optimizing. Instead of “Can I give this more cores?”, I ask which resource the slow phase is waiting for. Memory capacity determines whether the workload can fit. Memory bandwidth helps determine how quickly useful work can keep moving.
| Observation in the notes | What it established |
|---|---|
| 6 CPU cores · 64 GB RAM | A quantized 30B MoE was usable without a GPU |
| 8 → 12 threads · +22% prefill | Prompt processing benefited in that run |
| Decode stayed flat | More threads did not remove that bottleneck |
| Second instance · ~6 GB swap | The speculative-decoding trial was invalid |
The constraint became part of the design
The useful result was not “any big model works on any small computer.” It was a more precise understanding of which combinations are worth testing: sparse architectures, appropriate quantization, a deliberate context budget, and realistic concurrency.
If I repeat this on another machine, I want the exact model file, runtime revision, memory use, context size, prompt-processing time, decode rate, and response quality in the same record. A fast answer that lost the task is not a successful optimization.
Modest hardware makes assumptions expensive enough to notice. That is part of why I enjoy working with it. It forces the software, the model, and the machine into the same engineering conversation.