Local LLM Eval Farm:
24 Models, Zero Cloud

Executive Summary

We started with a question that sounds simple but isn't: on a single machine with 256GB of RAM, which local LLMs are actually worth running, and when?

Over two days, we pulled 24 models ranging from 2GB to 142GB, built a six-dimensional evaluation suite from scratch, discovered that the conventional wisdom about larger models being better is wrong for at least two important use cases, found a 5.8x throughput advantage sitting unused in the MLX backend, and shipped a four-tier cascade router backed by real eval data rather than guesswork.

Everything ran locally. Zero cloud API calls. Zero per-token cost. Unlimited iteration.

The Three Headline Findings

1. Size is not quality. qwen2.5:7b (5GB) scored 100% on multi-turn conversation. llama3.3:70b (42GB) scored 47.8% on the same eval, worse than the 3B model. For conversational tasks, running the 70B model is not just wasteful, it is actively worse.

2. The right backend matters as much as the right model. MLX and Ollama run the same model at the same quality. At 32 concurrent users and long outputs, MLX delivers 619 tok/s aggregate versus Ollama's 107 tok/s, a 5.8x difference that is entirely invisible unless you measure it.

3. One model wins on value. qwen2.5:7b is the answer to almost every question: 80.6% quality, 100% multi-turn, 93% domain, 10,601 tok/s peak throughput, 5GB. No other model comes close on value per gigabyte.

The Brief & Infrastructure

The Machine

Mac Studio with Apple M-series chip and 256GB unified RAM. This is not a GPU cluster, it is a single consumer machine that happens to have enough memory to fit every model we tested simultaneously if needed. The unified memory architecture means CPU and GPU share the same pool, which changes the calculus on model loading and concurrency compared to discrete GPU systems.

The Two Backends

We ran two model serving backends side by side:

• Ollama (port 11434), the de facto standard for local LLM serving. Manages model downloads, serves an OpenAI-compatible API, handles multiple models, and queues concurrent requests. Uses llama.cpp under the hood with GGUF quantized weights.
• MLX (port 8081). Apple's own ML framework optimized for Apple Silicon. Uses native Metal shaders. Runs safetensors weights. Has a built-in server (mlx_lm.server) with OpenAI-compatible API and native speculative decoding. Installed via pip install mlx-lm.

Models Pulled

24 models across 8 size classes, selected to span the useful range on a 256GB machine:

Evaluation Methodology

Six Eval Dimensions

We designed six distinct eval suites, each testing a different capability axis. Running a single combined eval would conflate very different failure modes, a model can be great at code and terrible at conversation. The suites are intentionally orthogonal.

Scoring Design Principles

• Deterministic where possible. Pattern matching, exact answers, and rubric-based scoring over LLM judgment for factual tasks. The multi-step math answer key was audited and corrected mid-run when we discovered the original key had wrong answers (A=24, B=12, C=19. Not A=20, B=10, C=15).
• External judge for open-ended tasks. llama3.3:70b judges open-ended responses. It never judges its own outputs. This matters, self-judging inflates scores by 8-15% in our testing.
• Checkpointed and resumable. Every eval saves a checkpoint after each model/task. A crashed run picks up exactly where it left off. This was essential, the full suite took ~18 hours of compute across both days.
• Serialized, never concurrent. Running multiple evals simultaneously causes GPU memory contention. qwen3 think models timed out at 300 seconds under any concurrency. Serializing runs added wall-clock time but produced reliable data.

The Scoring Bug Chronicle

Five scoring bugs were discovered and fixed mid-run across the eval scripts. Each required a partial re-run for affected models:

• sql_injection: 5-point task was being scored as 3 points. Fixed.
• off_by_one, had a spurious gate condition that rejected correct answers. Removed.
• clinical_stats, only credited full Bayes calculation, not the correct "base rate alone" shortcut. Fixed to accept either.
• fermi_estimation, rejected answers expressed as "/day" vs "/year." Fixed to normalize units.
• evolutionary_reasoning, required "inverted" when "non-inverted" is also a valid correct answer. Fixed.
• multistep_math, answer key in quality_eval_r2.py was wrong (A=20, B=10, C=15). All 22 models scored 0/10. Fixed to correct answers (A=24, B=12, C=19). All 22 models needed re-score on this task.

Quality Eval: 22 Models, 16 Tasks

The quality eval (Round 2, after scoring bug fixes) is the most comprehensive single dataset. 22 models, 16 tasks across four categories: reasoning, coding, knowledge, and instruction following.

Full Leaderboard

* qwen3:235b score heavily penalized by 300s timeouts on coding/reasoning tasks under concurrent eval load. Estimated real quality ~80%+.

The Three Structural Patterns

The knowledge ceiling at 67%. Every model except llama3.3:70b and deepseek-r1:32b (both 100%) hit the exact same 67% knowledge score. This is not a coincidence, it traces to two specific tasks. The hallucination_probe asks about Han Kang's 2024 Nobel Prize (most models have a training cutoff before October 2024 and correctly refuse). The confabulation_trap presents a fabricated Einstein quote and tests whether the model refuses to validate it. Models that score 100% on knowledge are the ones that refused cleanly on both. The 67% floor is structural, not meaningful.

Coding floor at 88%. With two exceptions (qwen3:8b and deepseek-r1:7b at 68%), every tested model scores 88-100% on coding tasks. Coding is the most saturated category, it is no longer a useful differentiator above 7B parameters.

Reasoning spreads widest (40-80%). The only category where model quality genuinely separates. Knights/knaves logic, Monty Hall, and logic grids are the hardest tasks in the suite. The 70B models do not win this category. qwen2.5:7b, qwen2.5-coder:14b, and qwen2.5:32b all tie at 80% reasoning alongside qwen2.5:72b.

Multi-Turn Eval: The Size Inversion

The multi-turn eval was the most surprising result of the entire project. We tested 6 models on 9 conversational scenarios using proper /api/chat endpoints with full message history. The results inverted everything the quality leaderboard suggested.

Results

The Four Patterns That Explain the Inversion

Pattern 1: Think mode cascades into failure. The gradual_refinement scenario asks models to iteratively improve code across 4 turns, add validation, add memoization, add docstring, return final version. qwen3 models in think mode spent 60-180 seconds per turn generating a reasoning chain, then often produced a subtly wrong intermediate. Turn 2 built on turn 1's error. By turn 4, the code was broken and the model was confident about it. This pattern explained nearly every qwen3 failure in the entire eval.

Pattern 2: Sycophancy scales with model size. In correction_handling, after giving a correct answer, the user pushes back with "I don't think that's right." Small models held their ground. Large models apologized and changed their answer. llama3.3:70b (the highest-quality model on static benchmarks) was the most sycophantic conversationalist. Trained on more human feedback, it learned too well that humans like it when you agree with them.

Pattern 3: instruction_persistence reveals stubbornness vs compliance. Models were told to always append a TL;DR after every response. Three scenarios later, they were explicitly told to stop. Most models acknowledged the request and then immediately appended "TL;DR: …" anyway. Only qwen2.5:7b stopped completely on the first ask.

Pattern 4: Context degradation is real above 12B. In long_context_degradation, models were given a list of 15 facts and asked questions drawn from early, middle, and late portions of the list across 6 turns. Models above 12B showed measurable accuracy drops on early-list facts by turn 6, the recency bias overwhelmed earlier context. qwen2.5:7b and llama3.2:3b showed no degradation.

Domain Eval: Medical, Legal, Math, Code, Science

8 models, 23 tasks, 5 high-stakes domains. This eval was designed for the T4 routing question: when a query needs expert-level reasoning, which model do you send it to?

Results

Five Domain-Specific Findings

Medical: everyone passes the floor. Even llama3.2:3b scored 86% on medical tasks. The medical_refuse task (a patient describes chest pain, respond appropriately) was passed by every model: all correctly advised calling emergency services rather than offering home remedies. Drug interaction tasks and clinical statistics (Bayes theorem with base rates) were handled correctly by all models above 7B.

qwen3 architecture dominates structured reasoning. qwen3:30b and qwen3:235b tied at 97.8%, perfect across medical, legal, math, and science. The think mode genuinely helps when the problem has a definite answer and there is no time pressure. Domain eval tasks are single-turn; the think mode cascading failure that hurt multi-turn evals is not triggered here.

llama3.3:70b science collapse (56%). The model that led the quality leaderboard at 83.8% failed on evolutionary reasoning and Fermi estimation. The science tasks that failed involve applying principles across domains rather than recalling established facts. This is a genuine capability gap, not a formatting issue.

deepseek-r1:14b math catastrophe (28%). Three of four math tasks failed. The reasoning chains were plausible but the final answers were wrong. The irrationality of √2 proof was abandoned mid-chain. This is likely a quantization artifact, the 4-bit GGUF weights clip precision in ways that cause mathematical reasoning to drift. The 32B version scores 80.6% on quality; the 14B is not a scaled-down version of that quality.

qwen2.5:7b legal weakness (75%). The only domain where qwen2.5:7b meaningfully underperforms. Contract ambiguity resolution and 4th Amendment analysis showed inconsistent reasoning across runs. Legal tasks require tracking multiple interpretive frameworks simultaneously, a task that benefits from the larger context that comes with bigger models.

MLX vs Ollama: The Hidden 5.8x

This was the most technically surprising result of the project. MLX and Ollama run the same model weights at the same quality. Benchmarking head-to-head with identical hardware, identical prompts, and increasing concurrency revealed a performance gap that is invisible at low concurrency and massive at any real multi-user load.

Aggregate Throughput vs Concurrency (qwen2.5:7b, long outputs)

Time to First Token at n=32

Why This Happens

Ollama serializes. When multiple requests arrive simultaneously, Ollama queues them and processes one at a time using llama.cpp. Aggregate throughput plateaus at roughly single-request performance (~107 tok/s for qwen2.5:7b) regardless of how many concurrent requests are sent. The 30th user waits for requests 1-29 to complete before getting their first token, hence 29s TTFT at n=32.

MLX batches natively. MLX's Metal backend processes multiple requests in a true batch on the GPU. Adding more concurrent requests increases GPU utilization without proportionally increasing latency. At n=32, MLX is using the hardware more efficiently than Ollama can at n=1.

When Ollama Wins

For very short outputs (fewer than 20 tokens) at concurrency 1, Ollama is slightly faster (1.5s TTFT vs 2.0s). The MLX batch setup cost is not recovered on trivial outputs. This informed the T1 routing decision: llama3.2:3b on Ollama for greetings and simple fact lookups, where the overhead matters and batching provides no benefit.

Model Council & Cascade Router

The eval data is only useful if it drives actual routing decisions. We built two systems: a Model Council for ensemble high-stakes responses, and a Cascade Router for single-model routing of every query.

The Cascade Router: 4 Tiers

The router classifies every incoming query into one of four tiers based on complexity signals, then routes to the appropriate model and backend. All assignments are backed by eval data.

Router Benchmark Results

The router was tested against 25 labeled queries representing all four tiers. Classification uses keyword patterns, complexity heuristics, and token budget estimation.

• 88% exact tier match on held-out test queries
• 100% within one tier, no query was routed two or more tiers away from correct
• Most common error: T3/T4 confusion on medical queries without explicit medical keywords (e.g., "explain how statins work" classified as T3 rather than T4)

The Model Council

For high-stakes queries where a single model answer is insufficient, the Council runs multiple models and synthesizes their outputs. Four modes:

• vote, each model answers independently; majority answer wins
• synthesize, llama3.3:70b reads all answers and produces a consensus
• debate, models argue positions; llama3.3:70b judges
• raw, all answers returned unfiltered for human review

Speculative Serving (Abandoned)

We implemented and tested a /speculate endpoint that uses qwen2.5:7b (T2) to draft tokens and qwen3:30b (T4) to verify them, the standard speculative decoding pattern. Results were negative: because the draft and verify models are so different in capability, the acceptance rate was low and the overhead of running two models outweighed the latency savings. Speculative decoding works well when draft and verify models are close in capability (e.g., 7B draft, 14B verify). The T2/T4 pairing is too dissimilar.

Surprises & Course Corrections

1. qwen3:235b Loaded Fine, and Was Nearly Useless

256GB of unified RAM means 142GB models load without complaint. qwen3:235b loaded in about 4 minutes and answered correctly when given time. The problem: "given time" means 90-300 seconds per query under any concurrency. During the concurrent quality eval, 4 of 8 coding and reasoning tasks hit the 300-second timeout and scored zero.

Its actual domain quality (97.8%) ties qwen3:30b. But qwen3:30b runs at 3,007 tok/s. The 235B model's real-world quality per second of wall time is the worst of any model tested. Having enough RAM to run it does not mean you should.

Lesson: Maximum RAM headroom is not an invitation to run the biggest model. Measure time-to-answer, not just answer quality.

2. The Answer Key Was Wrong

Every model in quality_eval_r2 scored 0/10 on multistep_math. After the first run, this was attributed to the models being bad at multi-step math. After the second run produced the same result, we audited the answer key. The key had A=20, B=10, C=15. The correct answers are A=24, B=12, C=19.

All 22 models were scoring correctly, the scoring script was grading them wrong. A full re-score pass was required.

Lesson: When every model scores 0% on a task, the task is broken, not the models. Universal failure is a red flag that demands auditing the rubric, not the responses.

3. The qwen3:14b no_think Anomaly

For qwen3:8b and qwen3:30b, think mode improves quality with acceptable latency tradeoffs. The conventional expectation is that think mode always helps on hard tasks, just at the cost of speed.

qwen3:14b inverts this: no_think scores higher on quality while running 2x faster. Think mode on the 14B model spends tokens on reasoning that consistently reaches worse conclusions than the model's direct answer. We don't have a mechanistic explanation, this is an empirical finding, not a theoretical one.

Policy derived: Always use no_think for qwen3:14b. Test think vs no_think empirically for any model; don't assume the documentation's recommendation matches your workload.

4. --decode-concurrency 8 Made Things Worse

The MLX documentation mentions --decode-concurrency as a tuning parameter for throughput. Setting it to 8 seemed like a reasonable optimization for a multi-user workload. The benchmark showed it was strictly worse than the default at every concurrency level above 4.

The dynamic batcher that MLX uses by default adapts to the actual batch size at runtime. A fixed concurrency setting of 8 creates a ceiling, once 8 requests are batched, subsequent requests wait even if GPU capacity exists. At n=32 concurrency, the fixed setting produces 320 tok/s versus the default's 619 tok/s.

Lesson: Dynamic scheduling outperforms static configuration for variable workloads. Measure before tuning. The default is often default for a reason.

5. Concurrent Eval Runs Corrupt Each Other

Early on, we attempted to run the quality eval and the concurrency stress test simultaneously to save wall-clock time. Both results were invalid: the quality eval showed anomalously high latencies, and the concurrency test had unusually low throughput. The unified memory bus does not partition, any workload on the machine affects every other workload.

Adding 6 hours of wall-clock time to serialize all eval runs produced clean, reproducible data. The time cost was real. The data quality improvement was essential.

Lesson: On a machine with unified memory, serialization is not optional for accurate benchmarking. Build checkpoint/resume into every eval from day one, you will need it.

Deliverables

Evaluation Scripts

Production Systems

Data Files

By the Numbers

Implementation Playbook

How to build a local LLM eval farm on Apple Silicon. Applicable to any Mac with 64GB+ unified RAM; scaled results with more RAM.

Step 1: Start With the Smallest Model That Works

Pull llama3.2:3b first. Run it at 512 concurrent requests. Understand what your machine can do at the ceiling before loading larger models. The smallest model establishes your throughput floor and your concurrency ceiling, everything else is measured against it.

Step 2: Build Checkpoint/Resume Into Every Eval From Day One

A full eval suite on 22 models takes 6-18 hours depending on model size and task complexity. Runs will crash. Models will timeout. The machine will be needed for other things. Every eval script should save state after each model response and resume from the last checkpoint. This is not optional, it is the difference between usable data and wasted compute.

Step 3: Serialize All Eval Runs

On Apple Silicon with unified memory, there is no isolation between workloads. Run one eval at a time. Build a queue if needed. Add 30-60% more wall-clock time to your estimate and accept it, the data quality difference is not subtle.

Step 4: Test Both Backends Before Committing

Install MLX (pip install mlx-lm) and run the same model on both Ollama and MLX with increasing concurrency. Do not assume one is faster, measure it. The answer depends on your concurrency pattern, output length distribution, and workload mix.

Step 5: Never Self-Judge

Use a separate model as judge. The difference between a model evaluating its own outputs versus a different model's outputs ranges from 8-15% score inflation. Designate your best available model as judge and never run it in the same eval it is scoring.

Step 6: Audit Tasks With Universal Failure

If every model fails a task, the task is broken. Check the rubric, the answer key, and the scoring logic before concluding that the models have a shared capability gap. Universal failure is a signal about the eval, not the models.

Step 7: Build the Router Last

Do not design routing rules before running evals. The router should be a consequence of the data, not a hypothesis that the data validates. Every tier assignment in the Cascade Router has a specific eval result behind it. If a tier assignment cannot be traced to a benchmark, it does not belong in the router.

Appendix: Full Data Tables

Master Model Table. All Dimensions

Think vs No-Think (qwen3 models)

RAG vs Bare. Full Results

Infrastructure Summary

Related Work

This eval farm powered the AI simulation infrastructure for the Talk Stories landing page project.