Choosing a local model: gemma3:4b vs qwen2.5:3b vs llama3.2:3b

What "small" means in 2026

In 2026, "small" refers to models between roughly 500 million and 2 billion parameters. Anything in this range fits comfortably in 4-8 GB of RAM when quantized to 4-bit, which is the format Ollama ships by default. The 1B-2B tier is the new entry point for phones, the 2B-4B tier is the sweet spot for laptops, and anything above 7B is firmly a workstation-class load.

The reason "small" works now is a combination of better training data and better post-training. The 2024 generation of small models (Phi-2, Gemma 1, Llama 2 7B) was clearly a tier below GPT-3.5 on long-form reasoning. The 2025-2026 generation (Gemma 3, Qwen 2.5, Llama 3.2) closes most of that gap on the specific tasks small models are actually good at: short-form summarization, code completion, JSON extraction, and routine writing.

The other reason is quantization. A "4B" model at Q4_K_M is roughly 2.5 GB on disk, fits in 4 GB of RAM, and runs at usable speed on any 4-core CPU from the last 5 years. The same model at Q8 is 4.5 GB on disk and noticeably slower but slightly higher quality. For a first-time setup, Q4 is the right default — bump to Q8 only if the Q4 model is producing obvious quality regressions on your workload.

Three models worth trying first

The table below summarises the three practical defaults. All three are available in the default Ollama registry, all three are redistributable, and all three speak the same HTTP API on port 11434.

Beyond the table, the practical advice is: try gemma3:4b first if you want a single default. It is the strongest of the three on code, reasoning, and multi-turn dialogue. Switch to qwen2.5:3b if you need faster responses or are running on very tight RAM. Switch to llama3.2:3b if you want the broadest community tooling — the Llama ecosystem has the most third-party fine-tunes and adapters by a wide margin.

All three can be installed at once. ollama list shows the downloaded models; ollama rm <name> removes one. Picking per-query is a matter of which tag the client passes to the API.

Benchmarks that matter

The two numbers that matter most for a local model are subjective quality and tokens-per-second on your hardware. MMLU, HumanEval, and MGSM scores are useful for ranking models against each other in the abstract, but they do not predict how a model will feel on a 6-year-old laptop with 8 GB of RAM. A model that scores 60 on MMLU but loses context after three messages is a worse daily driver than a model that scores 52 and stays coherent.

For subjective quality, the practical test is a 5-message conversation. If the model loses context between messages, misremembers a name, or hallucinates a function that does not exist, it is not ready for daily use regardless of its MMLU score. gemma3:4b is the strongest of the three on this test, with qwen2.5:3b close behind and llama3.2:3b noticeably behind on long dialogue.

For speed, the relevant numbers are tokens-per-second for prompt ingestion and tokens-per-second for generation. On a modern x86 CPU with AVX2, expect 20-40 tok/s for a 3B model and 12-25 tok/s for a 4B model at Q4_K_M. On Apple Silicon, M1 and newer hit 40-80 tok/s for 3B. On a recent Android phone, expect 8-15 tok/s for a 2B model and 5-10 tok/s for a 4B model. Below 8 tok/s, chat feels sluggish and the user starts reaching for the cancel button.

Context length is the third number worth knowing. gemma3:4b ships with 128K context, qwen2.5:3b with 32K, llama3.2:3b with 128K. Larger context costs more RAM and slows the model down — for a 128K context, expect 2-3x the latency of a 4K context on the same prompt. For chat and most code completion, 8K-16K is plenty.

How to switch models in Ollama

Ollama has three ways to pick a model. The simplest is ollama run:

ollama run gemma3:4b

This starts a chat session with that model. The model is pulled from the registry on first run and cached for next time. To pull without running, use ollama pull:

ollama pull qwen2.5:3b
ollama pull llama3.2:3b

Once the model is on disk, Ollama serves it via the HTTP API. The server listens on localhost:11434 by default. Any client that speaks the Ollama protocol can hit the API; the default model is the most recent one started, but the client can override per-request:

curl http://localhost:11434/api/generate -d '{
  "model": "gemma3:4b",
  "prompt": "Summarise the second paragraph of Moby-Dick."
}'

The OLLAMA_MODEL environment variable sets the default for any session that does not specify a model. To pin a model across reboots, add export OLLAMA_MODEL=gemma3:4b to the shell profile. The same value can be passed to the server directly:

OLLAMA_MODEL=qwen2.5:3b ollama serve

For batch jobs or scheduled scripts, the --model flag on ollama run is the right choice — it does not require editing the shell profile and does not affect other sessions.

If a model is not in the public registry, the Ollama Modelfile format wraps a custom GGUF. The minimum Modelfile is two lines:

FROM ./my-custom-weights.gguf
PARAMETER temperature 0.7

Then ollama create my-custom && ollama run my-custom brings it online with the same HTTP API and the same JSON payload shape.

When to upgrade your hardware

The right time to upgrade is when the model you want to run does not fit in the RAM you have, with a little headroom. The general rule is: 1.5-2x the model's on-disk size in free RAM. A 2.5 GB model wants 4-5 GB free, a 4 GB model wants 6-8 GB free, a 7 GB model wants 10-14 GB free. Headroom matters because the OS and the rest of the apps will compete for the same memory.

Under 8 GB of free RAM, stay with 2-3B models. The difference between a 2B and a 3B model on Q4 is roughly 1.5 GB of RAM and a noticeable quality gap. The difference between a 3B and a 4B model on Q4 is another 1 GB and a smaller quality gap. Diminishing returns set in fast under 8 GB.

8-16 GB of free RAM is the 4-7B sweet spot. The 4B tier is what most "small but capable" guides target. The 7B tier is the first tier where clear reasoning improvements show up, but 8 GB total is not enough for Chrome plus a 7B model — keep 16 GB in mind as the practical minimum for that tier.

16-32 GB of free RAM opens the 7-13B tier. These models are a clear step up on long-form reasoning, multi-step code, and structured output, but they are slow on CPU and require a recent GPU for real-time use. CUDA, Metal, and Vulkan paths all work in Ollama; the offload is automatic when a compatible GPU is detected.

32 GB or more puts fine-tuning on the table. LoRA on a 7B base model fits in 16 GB of VRAM; full fine-tuning of a 3B model fits in 24 GB. Both are out of scope for this guide, but the fact that they are reachable on consumer hardware is the most important shift in the local-model landscape of the last two years.

How Pulse handles this

Model switching uses the same path as any other Ollama-compatible client. To change the default, update the model path under ~/.ollama/models or set the OLLAMA_MODEL environment variable. Ollama exposes a stable HTTP API on port 11434, so the same JSON payload works from any client that speaks the Ollama protocol — no client-side model catalogue, no proprietary download path. The default for the first run is gemma3:4b; the user can swap to any installed model without restarting the service.