Local AI: the best open-weight LLMs for writing in English

Which local AI writes best?

If you want one recommendation rather than a methodology, this is it:

What counts as “local” here

Four conditions, applied strictly:

1. Open weights. The model files must be publicly downloadable and runnable outside the vendor’s infrastructure.
2. Genuinely runnable offline. Not a gated API with an open-sounding name.
3. Under 256 billion total parameters. Strictly below the threshold.
4. Already evaluated on this site. No estimates, no placeholder entries in the ranking itself.

The parameter limit applies to total parameters, including for mixture-of-experts models. This is worth stating plainly because it is a common and consequential error: an MoE model that activates a small fraction of its parameters per token still has to have all of its weights in memory. Total parameters determine eligibility and memory requirements; active parameters affect speed. When both counts are known, the model name keeps both (for example, 35B A3B) so the active count can inform speed expectations without being mistaken for a memory figure.

One thing the criteria do not require: that we generated the model’s responses on a machine in this office. What makes a model eligible is that the weights are available and the model can be run locally — not the historical location of the hardware that produced its texts. Where a variant’s throughput was measured on our own hardware, the speed figure is labelled as an AIWB measurement. Figures taken from elsewhere are labelled as external or estimated. We would rather show a labelled estimate than a confident invention.

Open source is not the same as open weights

“Open source” is the phrase people search for, so it is worth untangling. Open source is a licensing claim broader than merely publishing weights; how that framework applies to all model artefacts is still contested. Most models in this ranking publish their weights under licences that range from permissive to distinctly restrictive — commercial-use limits, acceptable-use clauses, redistribution conditions.

We rank on writing quality. The licence is a separate question, and an important one if you are shipping a product rather than drafting an email. Check it before you build on a model; it has no bearing on how well the model writes.

Eligible, but not yet tested

Models appear here only once they have been evaluated. Anything that meets the criteria but hasn’t been through the corpus yet sits in a short waiting list, without a score and without a rank, so you can see what’s coming rather than wonder whether we’ve noticed it.

Quantization, without the marketing

Quantization stores a model’s weights using fewer bits per value. A model trained in 16-bit precision can be repackaged at 8 bits, or 4, shrinking the download and, more importantly, the memory footprint. On the kind of inference most people run at home — one user, short batches, bottlenecked by how fast memory can be read rather than by raw compute — a smaller file often runs faster too, because there is simply less data to move per token.

It also changes the model. Sometimes barely; sometimes visibly.

What we can say with confidence is that the size of the effect is not a constant. It varies with the model, the quantization method, the task, the language, the context length and the level of quantization. This is why you will not find a “Q4 keeps 98% of quality” claim on this page. That number does not exist as a general fact, and repeating it is how people end up disappointed by a specific model on a specific task.

The tags you’ll see in the filter:

• Q8 — an eight-bit variant. We treat it as the practical high-quality reference point in this ranking: it is the version we reach for when memory allows. That is a working convention, not a claim that Q8 is mathematically equivalent to BF16 or FP16. Whether the remaining gap is measurable in English prose is an empirical question, and we prefer to answer it with paired tests rather than assertion.
• Q4 QAT — quantization-aware training. The model was prepared or trained with four-bit execution in mind, which makes it a distinct checkpoint rather than a post-hoc compression of an existing one.
• Q4_K_M — a four-bit GGUF quantization applied after training, with mixed precision across different parts of the model. This is not QAT, and conflating the two is a genuine source of confusion. Included by default.
• Q6 — a middle ground that appears when an evaluated variant exists.

Filters apply as you change them, and the state is carried in the URL, so a filtered view can be linked or bookmarked.

Muse Glimmer’s two K-Quant builds

Meta publishes Muse Glimmer 30B in two unusual GGUF builds that average approximately four bits per weight. K-Quant Dynamic keeps selected tensors at higher precision and targets 32 GB of VRAM; K-Quant 17GB compresses more aggressively and targets 24 GB. They are not Q4_K_M files, so we keep their exact names and score them as separate variants. Both sit under the broad Q4 filter because that filter groups practical four-bit-class downloads rather than claiming an identical encoding.

Meta reports average degradation of 0.2% and 1.0% respectively across fifteen common benchmarks. Those are publisher measurements, not AIWB writing scores and not guarantees for English, French, German or Spanish prose. Our rankings and direct comparison measure the two files independently on native writing prompts.

Hardware recommendations

Why bother running it locally at all

The cloud models at the top of our general leaderboard are strong. Local models are chosen for reasons that have nothing to do with beating them.

When the runtime and its integrations stay local, your text need not leave the machine — which is the whole argument for anyone drafting confidential, legal, medical or unpublished material. It works with no connection, on a plane or a bad hotel connection. It doesn’t change underneath you: a checkpoint you downloaded behaves the same next month, where a cloud endpoint can be updated, deprecated or retired without warning. There’s no per-request cost and no rate limit. And you can inspect, fine-tune and integrate it on your own terms.

Set against that: you are trading some quality for that control, and you’re providing the hardware. Whether the trade is worth it is exactly the question this ranking exists to inform.

Quality against speed

These are separate axes, and we keep them separate on purpose. The ranking column measures how well a variant writes English. The speed figures measure how fast it produces tokens on specific hardware. Nothing in this page combines them into a single “best” score, because the right balance depends on what you’re doing.

Drafting a long article, where you’ll read and edit the output carefully, rewards quality. Interactive back-and-forth — rephrasing, brainstorming, dialogue — rewards responsiveness, and a slightly weaker model that answers in two seconds may serve you better than a stronger one that takes thirty. Both variants are listed in every hardware tier for that reason.

FAQ

What is the best local AI for writing in English? The live summary above names the current top-ranked variant with its score and sample count. Bear in mind that the top variant is only the best answer if it fits your memory with room for your context; the per-tier recommendations above are more useful for most readers.

What does “local LLM” mean? An LLM is a large language model — the type of AI that generates text. A local LLM is one you download and run on your own computer, offline, rather than calling a service over the internet.

Why is there a 256B parameter limit? The threshold keeps this page centred on models that remain plausible on personal or compact workstation hardware. It applies to total parameters, including for mixture-of-experts models, because all the weights must be in memory even when only a fraction is active per token. Larger systems are outside this page’s scope rather than impossible in absolute terms.

Does quantization ruin quality? Not inherently, and not by a fixed amount. The effect depends on the model, the method, the task and the language. We show quantizations as separate ranked rows so you can see the difference where we’ve measured it, instead of trusting a general percentage.

What’s the difference between Q4 QAT and Q4_K_M? Q4 QAT is a distinct checkpoint prepared or trained for four-bit execution. Q4_K_M is a four-bit GGUF quantization applied to an existing model after training, using mixed precision across its layers. They are not interchangeable and we never merge them into one row.

How much VRAM do I need? More than the model file. Budget for the weights plus the runtime plus the KV cache, which grows with your context length. The dedicated GPU tab gives recommendations for common VRAM sizes with realistic headroom built in.

Is unified memory as good as VRAM? For capacity, often better — unified-memory machines can hold models no consumer graphics card can. For bandwidth, generally not, and that difference shows up as slower generation. Don’t treat the two as equivalent.

Are open-weight models open source? Not automatically. Publishing weights does not by itself establish an open-source licence, and model licences vary considerably in what they permit commercially. The licence is independent of writing quality — check it separately.

Why do these scores include comparisons against cloud models? Because it keeps every variant on one scale and preserves its full match history. The rank is local; the score comes from the general English leaderboard.

How often does this page update? Continuously. Rankings, filters and hardware recommendations are generated from live data, and new variants enter the ranking as soon as they’ve been evaluated on the English corpus.