Skip to content

Local Model Viability Factors for Coding

A local model is viable for coding only when RAM, model size, context length, and tool-calling reliability all clear the task's bar.

Run a local model for coding when three conditions hold at once: you have enough unified memory to fit the model plus its context, the task is small and well defined, and a privacy or offline requirement justifies the setup cost. Miss any one and a hosted frontier model is the better default. Local models became usable for some agentic coding in 2026, but the honest practitioner verdict is "hit and miss" and "not ready for a plug-and-play experience" (Böckeler, Thoughtworks).

Treat viability as a funnel, not a yes or no. A setup has to clear every stage in order: fit in RAM, respond at a bearable speed, call tools without stalling, produce functionally correct code, hold up across a longer conversation, and only then handle a larger task (Böckeler).

The factors that gate viability

Several factors interact, which is what makes local setups tedious to evaluate — one model gave better code on a stronger machine with identical settings, and manual and automated evals disagreed on which model was best (Böckeler).

Factor Why it gates viability
RAM / VRAM Model weights plus the context cache must fit, or the runtime crashes or crawls. A comfortable range was 15 to 25 GB of model on a 48 GB machine; a 48 GB model on a 64 GB machine crashed once the conversation continued (Böckeler).
Model size and architecture More parameters mean better output but a larger footprint. Mixture-of-experts models activate only part of their weights, so a 35B MoE needs less RAM and runs faster than a 35B dense model (Böckeler).
Quantization Compressing weights to Q4 or Q6 shrinks the footprint at some quality cost; quantization-aware training aims to preserve quality better (Böckeler).
Context window Runtime defaults are far too small for agentic loops; set at least 32K, ideally 64K. Larger context consumes more RAM through the key-value cache, so context ambition is capped by memory (Böckeler).
Tool calling Agentic coding needs schema-valid tool calls every turn. Models not fine-tuned for tool use emit malformed calls, though they often self-recover (Böckeler; XDA).
Harness overhead System prompt and tool schemas eat scarce local context. A lean harness such as OpenCode or Pi preserves more room than a heavier one (Böckeler).

Reasoning is not a free win. Smaller models often loop inside their chain of thought ("Wait… Actually… But wait…"); turning reasoning off ran faster and scored the same or slightly better in one automated eval (Böckeler).

Why it works

The setup is governed by a memory budget and a capability floor. Model weights plus the key-value cache, which grows with context length, must fit in physical memory; exceed it and the runtime crashes or thrashes. A fixed RAM ceiling therefore forces a three-way trade among model size, quantization, and usable context (Böckeler). Separately, agentic coding needs the model to emit valid tool calls turn after turn; models not trained for that stall the loop, which is why small local models suit autocomplete far better than agentic use (XDA). Mixture-of-experts architectures relax the memory side by activating a subset of parameters per token, which is why a roughly 35B MoE model hit the best capability-per-gigabyte balance in practice (Böckeler).

When this backfires

Local inference costs more than it returns under these conditions:

  • Under about 48 GB of memory, a 15 to 25 GB model plus a 32K to 64K cache will not fit comfortably, so the model crashes or crawls (Böckeler).
  • Complex or multi-file tasks blow up reasoning chains and context. Even a capable 80B model crashed on the next turn, and other runs stalled for 8 to 12 minutes before being abandoned (Böckeler).
  • With no privacy or compliance driver, the tuning time and the quality gap against hosted frontier models make hosted the rational choice.
  • A model that is not fine-tuned for tool calling derails the agentic loop with malformed calls (XDA).

The gap is narrowing, not closed. Independent evals report several local models now tool-call reliably, which reinforces that viability is a per-configuration question rather than a settled yes or no (Böckeler).

Example

After four weeks across an M3 Max (48 GB) and an M5 Pro (64 GB), Böckeler's go-to local setup was Qwen3.6 35B-A3B MoE at Q4 (about 22 GB), run in LM Studio with a 64K context window and a lean harness. It offered the best balance of capability, speed, and memory footprint among the models tried, and now handles small, well-defined day-to-day changes — while larger models either crashed or stalled (Böckeler, factors; experiences).

Key Takeaways

  • Local coding viability is a funnel of RAM fit, speed, tool calling, correctness, context length, and task complexity — a setup must clear every stage.
  • RAM is the hard constraint: weights plus the key-value cache must fit, which forces a trade among model size, quantization, and context length.
  • Mixture-of-experts models around 35B at Q4 give the best capability-per-gigabyte balance for a single developer machine today.
  • Reliable tool calling, not raw parameter count, gates agentic use; small models suit autocomplete better than agentic loops.
  • Choose local when a privacy or offline requirement justifies the tuning cost; otherwise a hosted frontier model is the better default.
Adopted — practitioner-sourced, corroborated Last reviewed: 2026-07-08
Feedback