Chikewa

Start here

Search

Category: AI & LLM

Run large language models and AI tools on your own hardware: llama.cpp, Ollama, LM Studio, GGUF models and local chat UIs — every setup tested in the lab.

  • GGUF and Quantization Explained: Pick the Right Model Size

    GGUF and Quantization Explained: Pick the Right Model Size

    Every local model you run is described by a name like some-model-8B-Q4_K_M.gguf, and almost everyone treats those bits after the name as magic strings. This guide takes them apart. What is a GGUF file, what do the two numbers in a model name actually mean, what a quantization level really costs in memory and quality, and how to pick one for the box you have. It is the reference the rest of the series points to, because once you understand size and quantization, “which model should I run” stops being a guess and becomes arithmetic.

    Beginner · 10 min · Concepts

    What a GGUF file is

    GGUF is a single-file format for storing a model. Inside one .gguf file you get the model’s weights, the architecture description, the chat template (the instructions that tell the model how to format a conversation), and the vocabulary. That single-file property is the whole point: there is no folder of parts to assemble, no separate tokenizer to match, nothing to forget. You copy one file from machine to machine, or download one file, and it runs. llama.cpp, Ollama and LM Studio all read it, which is why the model ecosystem is built on it. A model you download as GGUF for one of those tools is the same file that the others load — the format is the common currency, and the tools are just different front ends for it.

    The two numbers in a model name

    Take a name like model-8B-Q4_K_M. The 8B is the parameter count: roughly eight billion parameters. A parameter is a number the model learned, and the count is a coarse measure of how much it can hold in its “head”. More parameters generally means more capability, but also more memory and slower generation, all else equal. The Q4_K_M is the quantization, and it is the second, equally important half of the name. You will see the parameter count written as 0.6B, 1.5B, 4B, 8B, 14B, 32B, 70B and so on. As a rule of thumb for the “B” number: the smaller ones are fast and lightweight, the ones in the 4B to 14B range are the current sweet spot for home hardware, and the 32B and 70B models are the high end that want a real GPU or a lot of RAM.

    What quantization means

    A model’s parameters are originally stored at high precision, typically 16-bit or 32-bit numbers. Storing and computing on those is accurate but heavy. Quantization is the process of storing each parameter with fewer bits, trading a little precision for a lot less memory and (often) faster computation. The “Q” in the name stands for the number of bits per parameter: Q8 is 8 bits, Q4 is 4 bits, and so on. Going from 16 bits down to 4 bits shrinks the file to a quarter of its size. The cost is that each parameter is now an approximation of the original, and the model’s output quality degrades a little as you compress harder. Quantization is what makes running a 70B model at all possible on consumer hardware — at full precision it would need hundreds of gigabytes, but quantized it can fit in a fraction of that.

    Reading the quantization codes

    The codes you see in practice, from highest to lowest quality, are roughly:

    • F16 / F32 — full precision. The original, uncompressed values. Used for training and as a starting point; far too big to run locally.
    • Q8_0 — 8-bit. Very close to full precision in quality, and about half the size. The “safe” high-quality option when your memory allows it.
    • Q6_K — 6-bit, a “K-quant”. A good quality/size compromise.
    • Q5_K_M — 5-bit. Often the best balance for models you want to run well without huge memory.
    • Q4_K_M — 4-bit. The most common default and the one to understand. The “_K” means it is a mixed-precision K-quant: some parts of the model are kept at slightly higher precision than others, which preserves more quality than a flat 4-bit would. The “_M” is a size variant of that scheme. Q4_K_M is the quantization most model hosts offer first, and for good reason: it is small enough to run on modest hardware and good enough that the quality loss is often barely noticeable.

    There are also lower options (Q3, Q2 and below) that squeeze the model much smaller at a real quality cost, and they are worth using when the alternative is not being able to run the model at all. The practical guidance is: reach for the highest quantization your memory can hold. If a model’s Q5 does not fit but its Q4 does, run the Q4. Do not run a Q4 when your box could hold the Q5 — the extra quality is free if you have the memory.

    The memory math, with real numbers

    Here is the arithmetic that turns a model name into a memory requirement. The weight size in memory is approximately the parameter count times the bits per parameter, divided by 8 to get bytes. Worked out for a few real cases on the kind of hardware in this series:

    • A 4.65B model at Q4 is about 2.95 GB of weights. That is the exact model we ran in the lab: it loaded into a 16 GB box with room to spare for a 32k context, and generated at about 14 tokens per second on a 4-core CPU. The file was 2.95 GB on disk, and the loaded model fit comfortably with several gigabytes of RAM left over for the rest of the system.
    • A 0.6B model at Q4_K_M is about 0.5 GB. This is the model we used for the Ollama tests; it loaded and generated almost instantly, which is why starting with a small model is the right first move.

    Two things to add to the weight size. First, the context window costs memory on top of the weights, and it grows with the context length, so a 32k context needs more RAM than a 4k one for the same model. Second, if you have a GPU, you want the weights to fit in VRAM, not system RAM — an 8 GB card can hold roughly an 8B model at Q4 plus a modest context, which is why “8B at 4-bit” is the number people aim for on an 8 GB GPU. The whole exercise is the same everywhere: count the parameters, apply the bits, add the context, and check it fits the memory you actually have. If it does not, drop the quantization or the model size, not your expectations about the rest of the box.

    How to pick a quantization for your box

    A method that avoids most dead ends:

    1. Find your memory budget. Free system RAM for a CPU box, or VRAM for a GPU. This is the ceiling.
    2. Reserve room for context and the OS. Do not spend the entire budget on weights. Leave a couple of gigabytes for the context and the rest of the system, or you will be swapping.
    3. Pick the model size first, then the highest quant that fits. Decide how smart you need it to be (the “B” number), then take the best quantization your remaining memory allows. Q8 if you can, else Q6, Q5, Q4 in that order.
    4. Prefer a smaller model at a higher quant over a bigger one at a lower one, up to a point. A 7B at Q5 often beats a 13B at Q3, because the lower quant degrades quality faster than the extra parameters add it. There is a crossover where the bigger model wins, but for home hardware the smaller-and-cleaner choice is usually right.
    5. Measure, do not assume. Run the model, note the tokens per second, and judge the output on a few real tasks. The numbers tell you whether the quantization you picked is the right trade for your use.

    Common mistakes

    Picking by parameter count only. “70B is the best, I want 70B” is how people end up with a model that cannot load on their box. The parameter count and the quantization are one decision, not two. Always do the memory math before you commit to the size.

    Running a low quant when a higher one fits. If your box can hold the Q5, running the Q4 is leaving quality on the table for free. Check the memory, then take the best quant that fits.

    Setting a huge context on a small box. The context is real memory, and it is the quiet thing that pushes a “should fit” model into swapping. If a model loads but is mysteriously slow, the context length is the first thing to cut.

    Mixing a model and quantization from different sources without checking they match. The GGUF file already has its quantization baked in, so a mismatch is not really possible within a single file — but it is worth knowing that the quantization is a property of the file you download, not a setting you apply later. You cannot “quantize down” a file you already have without running a quantization tool on the full-precision weights.

    How this fits the rest of your home server

    Size and quantization are the vocabulary that connects every other guide in this series. When the llama.cpp guide tells you to check whether a model fits, this is the math it means. When the Ollama guide says to start with a small model, this is why 0.5 GB loads instantly and 2.95 GB is the sweet spot on a 16 GB box. When you are deciding whether a bigger model is worth it, the tokens-per-second number you measure is the answer to the question this guide sets up. The model file is the constant: pick the size and quantization that fit your memory, download it once as a GGUF, and any of the runtimes can load it. That is the entire decision, reduced to arithmetic and one measurement.

    Tested on:

    Hardware4-core / 16 GB
    4.65B model at Q4_K2.95 GB on disk, runs in 16 GB RAM with a 32k context
    0.6B model at Q4_K_M522 MB on disk, loads instantly

    Last tested: 15 September 2026

    What’s next?

    The natural next steps from this guide:

  • Run Local AI on Your Home Server: The Complete Guide

    Run Local AI on Your Home Server: The Complete Guide

    Local AI has crossed a line: you no longer need a data-centre’s worth of silicon to run a model that answers questions, drafts text and summarizes documents. On the same mini PC or old desktop you already use for your home server, you can run a large language model entirely on your own hardware — no API key, no per-token bill, no prompt ever leaving your network. This guide is the map for that whole territory. It tells you what your machine can realistically run, which of the three main runtimes (llama.cpp, Ollama and LM Studio) fits how you work, how to pick a model by its size and quantization, and how to put a proper chat interface in front of it. Every command and number below was checked on real lab hardware, and each section links out to the dedicated guide for that piece of the stack.

    Beginner · 12 min · Local AI

    Why people bother: a local model is private by default, works offline, costs nothing after the hardware, and you can swap models whenever you like. The trade-off is that the quality ceiling is set by your RAM and GPU, so the first honest question is not “which app” but “what can my box actually run”.

    What your hardware needs to run

    The single most important number is how much of a model’s weights fit in memory at once. A model’s on-disk size is a close proxy for the memory it will want when loaded, plus a margin for the context window you give it. As a working rule:

    • Roughly 4–8 GB of free memory runs small models (around 0.5B to 4B parameters at 4-bit) comfortably on CPU. These are fast and good enough for summarization, quick Q&A and drafts.
    • 8–16 GB opens up mid-size models (7B–14B at 4-bit). On CPU these are slower but usable for short tasks.
    • A GPU with 8+ GB of VRAM changes everything: it lets you load the model weights into fast video memory and generate at tens of tokens per second instead of a handful. This is the difference between “it answers” and “it feels instant”.

    In the lab we used a 4-core Intel i5-6500T with 16 GB of RAM and no discrete GPU. It ran a 4.65B-parameter model at 4-bit at about 48 tokens per second reading a prompt and 14 tokens per second generating. That is genuinely usable for everyday tasks — you type, and the answer streams back in a couple of seconds. The point is not that this box is powerful; it is that a modest box is enough to start, and you can always grow into a GPU later.

    If you are deciding what to buy specifically for this, the home server hardware guide has real benchmarks and the same “mini PC is the best all-rounder” conclusion applies: a used mini PC with 16 GB of RAM is the cheapest way into local AI that does not feel like a toy.

    The three runtimes, and when to use each

    Under the hood, all three popular tools are doing the same job: loading a GGUF model file and serving it. They differ in how much they hide and how much control they give you.

    • llama.cpp is the foundation. It is a C++ program you build once, and it gives you the most control and the best pure-CPU performance. You drive it from the command line. Choose it when you want maximum performance on a CPU box, you are comfortable in a terminal, or you want to tinker with context length, threads and quantization yourself.
    • Ollama wraps the same engine in a single install script and a tiny CLI plus an API. One command pulls a model from a registry; another runs it. It is the fastest way to “just have a local model” and the one most tools assume. Choose it for a low-maintenance, scriptable setup.
    • LM Studio is a desktop app with a graphical interface: browse a model catalogue, one-click download, chat in a window, and flip on a local server when an app needs the API. Choose it if you want the friendliest on-ramp and you are working on a machine with a screen rather than a headless server.

    The good news is that the model files are interchangeable across all three. A GGUF you download for Ollama can be loaded in llama.cpp, and vice versa. So pick the runtime for how you like to work, not out of fear of being locked in. A practical comparison of the three, with the numbers we measured, is in the llama.cpp vs Ollama vs LM Studio comparison.

    Choosing a model: size and quantization

    Every model you will run is described by two numbers: its parameter count (the “B” number — 0.6B, 4B, 8B, 14B, 70B) and its quantization (Q4_K_M, Q5_K_M, Q8_0, and so on). More parameters generally means smarter output; a higher quantization means less quality lost when the model is compressed to fit in memory. The two trade against each other on the same memory budget: a smaller model at a higher quant often outperforms a bigger model at a lower one, up to a point.

    As a starting point on a 16 GB box with no GPU, a 4B to 8B model at 4-bit is the sweet spot. It loads quickly, leaves room for a long context, and is fast enough to be pleasant. If you add an 8 GB GPU, an 8B to 14B model at 4- or 5-bit is the target. What each quantization level actually costs in memory and quality — including what “Q4_K_M” means rather than treating it as a magic string — is broken down in the GGUF and quantization explainer.

    Putting a chat interface in front of your model

    A raw API endpoint is powerful but unfriendly for day-to-day use. Most people want a chat window with conversations, a model picker and a place to paste documents. Open WebUI is the most popular self-hosted option: it runs in a Docker container, connects to Ollama (or any OpenAI-compatible endpoint) in a couple of lines of config, and gives you a clean, chat-app-style interface that works in a browser on any device on your network. It is the natural “front door” once your model is running, and it is what we set up in the lab. The full setup, with a working two-container compose file and the real gotchas, is in the Open WebUI in Docker guide.

    Keeping it safe on your network

    Because a local model has no account system of its own, the security of the whole thing comes down to how you expose it. The defaults are your friend: bind the model server and the web UI to 127.0.0.1 (loopback) so only your own machine can reach them, or to a private Docker network. To use the UI from your laptop or phone, put it behind your VPN rather than opening a port to the internet — Tailscale in Docker is the simplest way to reach a service on your home network from anywhere without punching holes in your router. If you do want the UI reachable across the house, the home server security guide covers firewalls and Docker network isolation, and Caddy in Docker shows how to front it with HTTPS if you need a proper domain. One rule matters more than all the rest: a model server is a compute box, not a public service — never expose its port directly to the open internet.

    A practical order to follow

    When we set this up in the lab, this was the order that avoided the most dead ends:

    1. Decide the budget. Check your free RAM and whether you have a GPU. This sets which model size you are realistic about.
    2. Pick the runtime. Terminal and control: llama.cpp. Low-maintenance and scriptable: Ollama. Desktop and graphical: LM Studio.
    3. Run one small model first. A 0.5B to 4B model at 4-bit. Get one real answer back before you spend time on anything bigger. This is the fastest way to prove the whole pipeline works.
    4. Measure it. Note tokens per second. Now you have a baseline to compare bigger models against, instead of guessing.
    5. Grow. Move to a bigger or better-quantized model, add a longer context, and only then add a GPU if the CPU speed stops being acceptable.
    6. Add the front door. Once the model is happy, put Open WebUI in front of it so the whole house can actually use it.

    Each of those steps is its own guide. Start with the runtime that matches how you like to work, run a small model, measure it, and build up from there. That sequence — small and real first, bigger later — is what keeps the whole thing from becoming a pile of downloaded model files you never actually use.

    How this fits the rest of your home server

    Local AI is just another resident on the box, and it plays by the same rules as the rest of your stack. It wants a fair share of RAM and, if you give it a GPU, the whole GPU while it is loaded; keep an eye on the box with the Prometheus and node_exporter setup so a chatty model does not quietly eat memory your media server needs. Back up the models folder and the runtime’s state with the same approach as the 3-2-1 backup strategy — a model is a few gigabytes of file, and re-downloading it is the only thing slower than losing it. And because the model lives on the server and the people using it live on their phones and laptops, the self-hosting starter guide is the right place to recap how all of this sits together on a machine you already run. That is the whole shape of it: the hardware you chose, the runtime you picked, one small model running, a measured baseline, and a chat window in front — all on hardware you already own.

    What’s next?

    The natural next steps from this guide: