Data · October 23, 2026

We checked our local-LLM speed estimates against 52 real benchmarks

Every machine page on this site shows a tokens-per-second number for each model. Those are estimates, not our own benchmarks, so we went and found 52 published runs to see how wrong we were. Short answer: pretty wrong on MoE models. Here's what we found, what we changed, and where we're still off.

26% → 5%median error, 43 fitted runs

12 → 37of 43 runs within ±15%

34% → 10%median error, 9 held-out runs

122% → 24%worst miss, fitted runs

Why simple bandwidth math goes wrong

The usual rule of thumb for local LLMs goes like this. To generate one token, the machine has to read every active weight from memory once, so tokens per second is roughly memory bandwidth divided by model size. That's what we used: 0.6 × bandwidth ÷ active weights, with the 0.6 standing in for "you never get the full spec-sheet bandwidth."

For big dense models that's in the right ballpark. It was actually a bit pessimistic. Llama 3.3 70B on Strix Halo measures 4.8 tok/s, and we said 3.6. Modern llama.cpp builds get closer to 80–90% of peak bandwidth on large reads, not 60%.

The real problem is mixture-of-experts (MoE) models. A model like gpt-oss-120b only touches about 5B parameters per token, so the formula sees a tiny read and predicts huge speeds. On an RTX PRO 6000 it predicted 397 tok/s for gpt-oss-120b. The measured number is 196. On an M3 Ultra, gpt-oss-20b came out at 257 predicted against 116 measured. That's more than 2× too high.

What the formula misses is that every token also has a fixed cost that doesn't scale with model size: launching GPU kernels, attention, and for MoE models, routing each token to its experts. When the weight read only takes a millisecond or two, that fixed cost is a big share of the time. Then a faster GPU barely helps, because it only speeds up the part that was already small.

The new formula, in plain words

time per token = active weights ÷ (efficiency × bandwidth) + fixed overhead
tokens/sec = 1 ÷ time per token

So there are two knobs per platform. Efficiency is how much of the spec-sheet bandwidth you actually get on big reads. Overhead is the fixed cost per token, with one value for dense models and a larger one for MoE. We fitted both to the benchmarks below. For each platform we picked the values that minimise the error in log terms, so a 20% miss on a slow run counts the same as a 20% miss on a fast one.

Platform	Bandwidth efficiency	Overhead, dense	Overhead, MoE	Runs fitted
NVIDIA GeForce / RTX PRO (CUDA)	83%	1.0 ms	2.5 ms	12
DGX Spark / GB10 (CUDA)	77%	0.0 ms	3.4 ms	7
AMD Strix Halo (Vulkan / ROCm)	80%	0.1 ms	5.1 ms	10
Apple M Pro / Max (Metal / MLX)	90%	1.2 ms	5.3 ms	8
Apple M Ultra (Metal / MLX)	84%	1.8 ms	6.5 ms	6

A few things stand out. NVIDIA cards have the smallest MoE overhead (2.5 ms), which is why they keep the lead on small-active MoE models. Apple and Strix Halo pay roughly twice that, around 5–6 ms per token. The M Ultra chips also lose a bit of efficiency compared with a single-die M Max. That fits the common report that two fused dies don't quite give you double the speed.

Before and after

On the 43 runs we fitted to, the median error dropped from 26% to 5%. 37 of 43 now land within ±15%, up from 12. The worst miss went from 122% to 24%.

Fitting to the data and then grading yourself on the same data flatters the result. So we also kept 9 runs out of the fit and only checked them afterwards. On those, the median error went from 34% to 10%, and 7 of 9 are within ±15% (the old formula got 0). That held-out set includes Jeff Geerling's Ollama runs of a 70B model on a Framework Desktop, a Dell GB10 and an M3 Ultra, plus the Strix Halo result that started all this: Qwen3.6 35B-A3B measured at 63 tok/s, where we used to say 84.5 and now say 71.5.

Each vertical line connects the old estimate (ring) to the new one (dot) for the same run. Hover a point for details.

What changed on the site

Here's how the numbers on a few machine pages moved, at 4-bit. Big dense models went up by about a third. Small-active MoE models on fast hardware came down a lot.

Machine	Model	Old est.	New est.	Change
NVIDIA GeForce RTX 5090 Founders Edition	Qwen3.6 35B-A3B	591	269	−55%
PNY NVIDIA RTX PRO 6000 Blackwell	GPT-OSS 120B	397	231	−42%
Mac Studio M5 Ultra	GPT-OSS 120B	266	109	−59%
Mac Studio M5 Max	Qwen3.6 35B-A3B	203	116	−43%
GMKtec EVO-X2	Qwen3.6 35B-A3B	84	72	−15%
NVIDIA DGX Spark	GPT-OSS 120B	60	61	+2%
GMKtec EVO-X2	Llama 3.3 70B	3.6	4.8	+33%
PNY NVIDIA RTX PRO 6000 Blackwell	Llama 3.3 70B	25	34	+34%

Where we're still off

Qwen3-Next / Coder-Next. These hybrid-attention models run slower than their 3B active parameters suggest. On Strix Halo we're still about 24% too high. Their attention layers do extra work that our formula doesn't model.
RTX 5090 with special builds. llama.cpp has an opt-in CUDA graph flag (GGML_CUDA_GRAPH_OPT=1) that cuts the per-token overhead a lot. With it, the 5090 runs small MoE models 26–37% faster than we estimate. We model default builds, since that's what most people run.
A few big MoE models. Qwen3.5-122B on DGX Spark (+20%) and gpt-oss-120b on M2 Ultra and RTX PRO 6000 (+18–19%) are still a bit optimistic. Qwen3-Coder-30B at Q8 on Spark is 18% pessimistic.
Benchmarks disagree with each other. The same Qwen3 30B-A3B on Strix Halo shows 75 tok/s in one source and 85 in another. That's about ±10% of noise from drivers, builds and settings, so being closer than that doesn't mean much.
Short context, one user. All of this is single-user decode at short context with llama.cpp-class runtimes. At 32K context, expect 20–40% slower. Speculative decoding, MTP and batching can be faster. Prompt processing (prefill) is a different story: it depends on compute, and NVIDIA pulls well ahead there.
No M1 Ultra. It came in far below any bandwidth-based model, and we don't list it, so we left it out.

All 52 benchmark runs

Every row links to where the number came from. Errors within ±15% are green. Rows marked held out weren't used to fit the formula. Click a column header to sort.

									Source
1	DGX Spark 273 GB/s	Qwen2.5-Coder 7B Q8_0 llama.cpp tg32	Dense	29.4	20.2	−31%	26.0	−12%	source
2	DGX Spark 273 GB/s	Gemma 3 4B Q4_0 llama.cpp tg32	Dense	81.0	64.9	−20%	83.4	+3%	source
3	DGX Spark 273 GB/s	gpt-oss-120b MXFP4 llama.cpp tg32	MoE	58.7	60.5	+3%	61.4	+5%	source
4	DGX Spark 273 GB/s	gpt-oss-20b MXFP4 llama.cpp tg32	MoE	83.4	85.6	+3%	80.0	−4%	source
5	DGX Spark 273 GB/s	Qwen3-Coder 30B-A3B Q8_0 llama.cpp tg32	MoE	61.1	46.7	−23%	49.8	−18%	source
6	DGX Spark 273 GB/s	GLM-4.7-Flash Q8_0 llama.cpp tg32	MoE	48.7	51.4	+6%	53.9	+11%	source
7	DGX Spark 273 GB/s	Qwen3.5-122B-A10B Q5_K_S llama.cpp tg128	MoE	23.0	23.8	+4%	27.7	+20%	source
8	Strix Halo 256 GB/s	Llama-2 7B Q4_0 llama.cpp Vulkan tg128	Dense	52.7	40.2	−24%	53.4	+1%	source
9	Strix Halo 256 GB/s	Qwen3 8B Q4_K_M llama.cpp Vulkan tg128	Dense	41.4	30.6	−26%	40.7	−2%	source
10	Strix Halo 256 GB/s	Mistral Small 24B Q4_K_M llama.cpp Vulkan tg128	Dense	14.3	10.7	−25%	14.3	0%	source
11	Strix Halo 256 GB/s	Llama 3.3 70B Q4_K_M llama.cpp Vulkan tg128	Dense	4.8	3.6	−25%	4.8	0%	source
12	Strix Halo 256 GB/s	Qwen3 30B-A3B Q4_K_M llama.cpp Vulkan tg128	MoE	85.5	76.8	−10%	67.3	−21%	source
13	Strix Halo 256 GB/s	Qwen3 30B-A3B UD-Q4_K_XL llama.cpp Vulkan tg128	MoE	75.3	76.8	+2%	67.3	−11%	source
14	Strix Halo 256 GB/s	Qwen3-Coder-Next 80B-A3B Q4_K_M llama.cpp Vulkan tg128	MoE	57.5	84.5	+47%	71.5	+24%	source
15	Strix Halo 256 GB/s	Llama 4 Scout UD-Q4_K_XL llama.cpp Vulkan tg128	MoE	20.2	14.9	−26%	18.0	−11%	source
16	Strix Halo 256 GB/s	gpt-oss-120b MXFP4 llama.cpp ROCm tg128	MoE	47.5	56.7	+19%	54.6	+15%	source
17	Strix Halo 256 GB/s	Qwen3 235B-A22B IQ3_XS llama.cpp Vulkan tg128	MoE	18.1	16.9	−6%	20.2	+12%	source
18	M5 Max 40c 614 GB/s	Llama-2 7B Q4_0 llama.cpp tg128	Dense	119.9	96.5	−19%	123.4	+3%	source
19	M5 Max 40c 614 GB/s	Llama-2 7B Q8_0 llama.cpp tg128	Dense	72.4	51.4	−29%	70.6	−2%	source
20	M5 Pro 307 GB/s	Llama-2 7B Q4_0 llama.cpp tg128	Dense	67.6	48.3	−29%	66.6	−1%	source
21	M5 Pro 307 GB/s	Llama-2 7B Q8_0 llama.cpp tg128	Dense	38.9	25.7	−34%	36.9	−5%	source
22	M5 Max 40c 614 GB/s	Qwen3.6-27B Q8_0 llama.cpp decode@128	Dense	18.0	12.9	−28%	18.9	+5%	source
23	M5 Max 40c 614 GB/s	Qwen3.6-35B-A3B Q8_0 llama.cpp decode@128	MoE	93.0	115.6	+24%	90.3	−3%	source
24	M5 Max 40c 614 GB/s	gpt-oss-120b MXFP4 llama.cpp 4K ctx	MoE	87.9	136.0	+55%	98.0	+12%	source
25	M4 Max 40c 546 GB/s	gpt-oss-20b MXFP4 llama.cpp tg128	MoE	117.8	171.3	+45%	108.8	−8%	source
26	M5 Ultra 1,228 GB/s	Llama-2 7B Q4_0 llama.cpp tg128	Dense	179.1	193.1	+8%	181.8	+2%	source
27	M5 Ultra 1,228 GB/s	Llama-2 7B Q8_0 llama.cpp tg128	Dense	119.0	102.9	−14%	114.4	−4%	source
28	M3 Ultra 80c 819 GB/s	Llama 3.3 70B 4-bit MLX (oMLX) 1K ctx	Dense	17.0	12.4	−27%	16.8	−1%	source
29	M3 Ultra 80c 819 GB/s	gpt-oss-20b MXFP4 llama.cpp tg128	MoE	115.5	256.9	+122%	107.8	−7%	source
30	M2 Ultra 76c 800 GB/s	gpt-oss-120b MXFP4 llama.cpp tg128	MoE	79.7	177.2	+122%	95.0	+19%	source
31	M2 Ultra 76c 800 GB/s	gpt-oss-20b MXFP4 llama.cpp tg128	MoE	116.1	251.0	+116%	107.0	−8%	source
32	RTX PRO 6000 1,792 GB/s	Llama-2 7B Q4_0 llama.cpp tg128	Dense	274.2	281.7	+3%	280.4	+2%	source
33	RTX PRO 6000 1,792 GB/s	Llama 3.3 70B Q4_K_M llama.cpp tg128	Dense	34.0	25.3	−26%	33.8	−1%	source
34	RTX PRO 6000 1,792 GB/s	gpt-oss-120b MXFP4 llama.cpp tg128	MoE	196.3	396.8	+102%	231.4	+18%	source
35	RTX PRO 6000 1,792 GB/s	gpt-oss-20b MXFP4 llama.cpp tg128	MoE	286.9	562.2	+96%	264.1	−8%	source
36	RTX 5090 1,792 GB/s	Llama-2 7B Q4_0 llama.cpp tg128	Dense	290.0	281.7	−3%	280.4	−3%	source
37	RTX 5090 1,792 GB/s	Qwen3.6-27B Q4_K_M llama.cpp tg128	Dense	77.1	64.0	−17%	81.3	+5%	source
38	RTX 5090 1,792 GB/s	Qwen3.6-35B-A3B UD-Q4_K_M llama.cpp tg128	MoE	271.0	591.2	+118%	268.6	−1%	source
39	RTX 5090 1,792 GB/s	gpt-oss-20b MXFP4 llama.cpp tg128	MoE	282.5	562.2	+99%	264.1	−6%	source
40	RTX 4090 1,008 GB/s	Llama-2 7B Q4_0 llama.cpp tg128	Dense	186.2	158.5	−15%	179.8	−3%	source
41	RTX 4090 1,008 GB/s	Qwen3 32B Q4_K_M llama.cpp tg128	Dense	42.0	30.4	−28%	40.4	−4%	source
42	RTX 4090 1,008 GB/s	gpt-oss-20b MXFP4 llama.cpp tg128	MoE	222.0	316.2	+42%	208.9	−6%	source
43	RTX 4090 1,008 GB/s	Qwen3 30B-A3B Q4_K_M llama.cpp tg128 (fusion off)	MoE	198.4	302.3	+52%	204.4	+3%	source
44	Strix Halo 256 GB/s	Qwen3.6 35B-A3B Q4_K_Mheld out Ollama/llama.cpp	MoE	63.0	84.5	+34%	71.5	+14%	source
45	M5 Ultra 1,200 GB/s	gpt-oss-120b MXFP4held out MLX short prompt (concurrency caveat)	MoE	99.0	265.7	+168%	108.8	+10%	source
46	M5 Max 40c 614 GB/s	gpt-oss-120b MXFP4held out MLX (oMLX) 1K ctx	MoE	92.8	136.0	+47%	98.0	+6%	source
47	RTX PRO 6000 1,792 GB/s	gpt-oss-120b MXFP4held out llama.cpp 4K ctx (newer build)	MoE	221.3	396.8	+79%	231.4	+5%	source
48	RTX 5090 1,792 GB/s	Qwen3 30B-A3B Q4_K_Mheld out llama.cpp + GGML_CUDA_GRAPH_OPT=1	MoE	352.1	537.4	+53%	260.1	−26%	source
49	RTX 5090 1,792 GB/s	gpt-oss-20b MXFP4held out llama.cpp + GGML_CUDA_GRAPH_OPT=1	MoE	419.1	562.2	+34%	264.1	−37%	source
50	Framework Desktop (Strix Halo) 256 GB/s	Llama 3.1 70B Q4_K_Mheld out Ollama (Geerling)	Dense	5.0	3.6	−27%	4.8	−3%	source
51	Dell Pro Max GB10 273 GB/s	Llama 3.1 70B Q4_K_Mheld out Ollama (Geerling)	Dense	4.7	3.9	−18%	4.9	+5%	source
52	M3 Ultra 512GB 819 GB/s	Llama 3.1 70B Q4_K_Mheld out Ollama (Geerling)	Dense	14.1	11.6	−18%	15.7	+12%	source

"Active GB per token" is active parameters × bits per weight ÷ 8. For dense GGUF files we back-solved the bits per weight from the file size. MoE rows use the same bits-per-weight values the rest of the site uses (Q4_K_M 4.85, Q8_0 8.5, MXFP4 4.25). M2 and M3 Ultra runs were used for the Ultra class at their own bandwidth. Spotted a benchmark we should add, or a mistake? Email hello@bestlocalairig.com. Full method: How we estimate.

See the numbers for your machine

Every machine page now uses the new formula. Pick a rig to see which open models fit and how fast they should run, or check the models page for memory needs.

Compare all rigs Models & memory needed Get the weekly email

We send one short email a week with price drops, restocks and new open models. If we re-run this check with new benchmarks, that's where it'll go first.