Choosing a Local Coding Model on a 32 GB Mac

The local model eventually produced a correct binary-tree inversion. Waiting for it was the problem.

It could load, accept a long context, and call tools. But a small coding task still took long enough to interrupt the work around it. Reducing the model’s precision saved memory without delivering the speed increase we expected.

That led us to build llamacpp-recipes: an open-source collection of serving presets, benchmark helpers, and Pi configuration examples for local inference.

The project uses upstream llama.cpp. It does not introduce another inference engine or model manager. Its purpose is to make a practical question easier to answer:

Which model and settings work well on the machine you actually have?

A model can fit in memory and still feel slow. A smaller quantization can use less RAM without generating noticeably faster. A setting that wins on a short prompt can lose once a conversation grows.

For a coding assistant, three measurements matter:

• Time to first token: how long the model spends loading or processing the prompt before it starts responding.
• Decode speed: how quickly it generates the response once that work is done.
• Time to a correct result: the whole task, including tool calls, mistakes, and repairs.

The last one is easy to overlook. A fast response that writes the wrong function creates more work.

We used a small BFS coding task alongside throughput measurements, then checked the generated implementations independently. That does not establish broad coding ability, but it stops “the model said it was done” from becoming the test result.

The test machine was an Apple M2 Pro with 32 GiB of unified memory, running macOS 27.0.1. Inference used llama.cpp with the Metal backend, not MLX.

We pinned the runner version and compared seven Unsloth MoE quantizations, with a dense Qwen3.8 model as a control. Every configuration used Q8 key/value caches and a configured context capacity of 262,144 tokens.

Here are three useful configurations from the October 2026 run:

These are decode rates, not end-to-end response times. Each cell averages four responses across two repetitions, using uncached and repeated-prefix requests. The longer prompts were synthetic ledger text, and each response generated 128 tokens.

We configured 256K capacity; we did not fill it. The largest measured input-plus-output occupancy was 20,624 tokens. Other desktop activity was not isolated, so these are observations from a working Mac, not laboratory ceilings.

The full report includes every tested quantization, memory observations, failed cases, and reproduction details.

Multi-token prediction, or MTP, proposes tokens that the main model verifies. When enough proposals are accepted at a low enough cost, generation gets faster.

That qualification matters.

Nemotron IQ2 reached 47.6 tok/s with MTP4 on the short prompt. At roughly 20K input, the same setting managed 23.4 tok/s, while turning MTP off produced 32.4 tok/s.

Qwen Q3 behaved differently: MTP4 retained a useful advantage at 20K. Gemma’s MTP2 gain at that length was small.

There were also failures. Nemotron Q4 with MTP2 or MTP4 ran out of Metal allocation capacity in our configuration. Its non-speculative preset worked, so the public examples include that mode rather than presenting the failed settings as usable.

The lesson is to compare speculation on and off for each model and workload. A high acceptance rate or a promising model card is not a substitute for that measurement.

Short prompts make attractive benchmark numbers. They can also hide the delay that matters most.

Qwen Q3 with MTP4 reached 58.3 tok/s on our short screen. With approximately 20K input, it generated at 22.7 tok/s and took about 54 seconds to produce the first token of an uncached request.

Gemma’s uncached 20K request took about 89 seconds to reach its first token. Nemotron IQ2 without MTP took about 49 seconds.

Repeated prompts were much quicker to start because the server reused their matching prefixes. That is useful for an agent making successive tool calls, but it does not make a new conversation equally fast.

A useful comparison should therefore include:

1. A short prompt to check basic operation and decode speed.
2. A prompt close to the size of the intended system instructions, tools, and working context.
3. A repeated-prefix request to check cache reuse.
4. A small real task with an independently checked result.

The repository’s comparison helper records native token counts and cache statistics, rather than estimating input length from characters. It also retains failures instead of quietly reducing context or changing precision.

Once you have a few useful candidates, restarting a manually configured server for every switch becomes tedious.

llama.cpp’s native router solves that part. It accepts a model ID with each request, loads the corresponding preset when needed, and forwards the request to that model.

Our launcher limits it to one resident model at a time. An idle model can be unloaded when another is requested; a busy model is allowed to finish while the new request waits.

You still choose the model. The router does not inspect the task and decide which model is best.

There is also a cost to switching: unloading discards the model’s in-memory KV cache. Returning to it may require processing the conversation again. For a long coding session, staying with one suitable model can be more efficient than changing models every few turns.

Start with the repository and its setup instructions:

git clone https://github.com/Chrysent-AI/llamacpp-recipes.git
cd llamacpp-recipes
make build

Download the selected GGUF files separately. Some configurations also need a matching vision projector or a separate MTP drafter.

The public example.ini contains the measured presets with generic relative paths. Copy the shared settings and only the model sections you intend to use into your private local/models.ini, then point those entries at your local files.

The serving helper does not download missing models.

make doctor
make dry-run
make serve

make doctor checks the native configuration and lists available model IDs without deliberately loading their weights. make serve starts the router from the private INI file; a model-path environment variable is not required.

The supplied listener is local-only. It is not an authenticated shared service, and it should not be exposed to a network without a separate security decision.

Pi can offer all these choices through one provider URL, with a separate model entry for each preset.

The repository includes a complete models.json example. Merge the entries you need into your existing Pi configuration rather than replacing unrelated providers.

Then open /model in Pi and select the matching model ID.

Each entry has its own sampling and thinking settings. This is important because Pi sends those parameters with the request: a temperature supplied by Pi overrides the server’s sampling default. Changing only the INI file will not change a request that supplies another value.

The examples also keep three controls separate:

• Context capacity: how much input and output the running model can accommodate.
• Response allowance: how many tokens a response may generate, including thinking.
• Thinking budget: the allowance for a thinking block, where supported.

MTP is another independent setting. A model ID ending in d0 means speculation is off; it does not mean thinking is off.

Image input is declared only for configurations that passed a basic projector-backed image check. Nemotron remains text-only in these examples.

For our measured setup, Qwen3.6 Q3 with MTP4 is a reasonable coding-and-vision starting point. Gemma QAT with MTP2 is another vision-capable option. Nemotron IQ2 without MTP was fastest on the longer text-only workload we tested.

All eight substantive Pi BFS trials produced correct implementations under independent checks. Those trials used isolated sessions with read, write, and edit tools—not Bash, extensions, or a full interactive harness. Their wall times excluded model loading and the independent tests. One small task is not a general quality ranking.

That is the intended use of these results: a shortlist and a reproducible method, not a universal winner.

Try the models on a few tasks you recognize. Keep the context, sampling, and cache conditions visible. Count repairs as part of the cost. If a configuration is fast but unreliable, that should appear in the result just as clearly as its tokens per second.

The code, public presets, Pi examples, and benchmark report are available in Chrysent’s MIT-licensed llamacpp-recipes repository. Model weights and upstream components retain their own licenses.