Chikewa

Start here

Search

Tag: LLM

A large language model: text-generation AI that runs locally or in the cloud.

  • Can a Mini PC Run a Local AI Model?

    Can a Mini PC Run a Local AI Model?

    A local LLM on a mini PC is one of the most attractive “what can this little box actually do” questions in homelabbing, because the answer used to be an obvious no. This article answers it with a real test: we ran an 8-billion-parameter model on the same modest hardware a typical 16 GB mini PC would have, and measured whether it fits and how it actually feels. No GPU, no cloud, no — just the box and the numbers.

    Intermediate · 8 min · Hardware

    What “run an LLM” requires, in one line

    The model’s weights have to fit in memory, and then the CPU (or a GPU, if you have one) has to compute. The weights are the predictable part — for a 4-bit model the file size is almost exactly the memory it takes. An 8B model at 4-bit is about 5 GB of weights, so it fits in a 16 GB box with room to spare for the operating system, your other containers, and a healthy context window. That is the short answer: yes, a 16 GB mini PC runs an 8B model.

    But “runs” and “feels good” are different, and on a CPU-only box that gap is the whole story. The more you want the model to be smart, the more it has to compute per token, and a small CPU has a hard ceiling on how many tokens it can produce per second.

    What we measured on a CPU-only box

    We tested on a 4-core Intel i5-6500T with 16 GB of RAM and no discrete GPU — deliberately an older, modest CPU, because that is the honest lower bound for “a cheap mini PC.” We ran three Qwen3 models at 4-bit and measured both memory and real speed:

    Model (Q4_K_M)Weights in RAMGeneration speedPractical feel
    Qwen3 0.6B~0.5 GB~47 tokens/sNearly instant
    Qwen3 4B~2.3 GB~9 tokens/sComfortable everyday use
    Qwen3 8B~4.9 GB~5 tokens/sUsable, but you wait on long answers

    Two conclusions. First, all three fit in 16 GB with room to spare, so memory is not what stops you from running a bigger model on a mini PC. Second, the 8B model is the boundary of “comfortable” on a CPU: it answers, it is genuinely useful for summarising and drafting and Q&A, but at about five tokens per second a long reply takes a noticeable while. The 4B model is where CPU-only local AI feels good on hardware like this.

    The deeper point is that a mini PC is the right shape of machine for this, even though it is not fast. It uses a little power, it is quiet, it runs the OS and your other Docker services in the same 16 GB, and the model lives in the headroom. You are not buying a GPU; you are buying a box that runs the whole server and a useful model in the margin.

    When the answer becomes a stronger yes

    Three things move a mini PC from “runs an 8B model” to “runs local AI that feels good”:

    • A faster CPU. A modern N100 or a mini PC with more, faster cores raises the token rate meaningfully, because CPU generation speed scales with the cores available. The 8B model that took ~0.2s per token on our 2015 four-core box will be noticeably snappier on current hardware.
    • A GPU. This is the big one. A discrete GPU with 8 GB+ of VRAM moves the weights into fast memory and lifts the generation rate to tens of tokens per second, which is the difference between “it answers” and “it feels instant.” If local AI is a primary use, a mini PC with a modest GPU is the step up.
    • Picking the right size. For CPU-only, a 4B model at 4-bit is the sweet spot — fast enough to feel responsive, smart enough to be useful. Reach for 8B when you need the quality and can accept the slower generation.

    The how-much-RAM-a-local-LLM-needs article in this cluster has the full measured table and the formula for sizing any model to any box, and the run local AI on your home server guide covers the runtimes (llama.cpp, Ollama, LM Studio) you would actually use on the mini PC.

    Power and cost: the quiet advantage

    One reason a mini PC is the natural home for a local model is the cost side. A small box idles at a fraction of what a GPU workstation draws, and local inference on CPU draws modestly more than idle. The total cost of a useful local model is the electricity of a box that was already running your media server and your other services — the model is an add-on, not a second machine. We do not have a validated power figure for this specific configuration to quote, so treat the exact wattage as something to measure on your own box rather than assume; the structural point (a model living in the headroom of an already-running box) holds regardless.

    Bottom line

    Can a mini PC run a local LLM? Yes — a 16 GB mini PC runs an 8B model comfortably in memory, and a 4B model runs it in a way that feels good on CPU. The honest caveat is speed: on a CPU-only box, the model’s intelligence and its responsiveness pull in opposite directions, and 4B is where the balance lands for most people. If you want an 8B model to feel instant, the upgrade is a faster CPU or a GPU, not more RAM. For the Chikewa reader who already runs a home server, the mini PC is the practical machine for local AI: it fits in the same 16 GB as everything else, it is quiet and cheap to run, and the numbers above tell you exactly what to expect before you buy it.

    Tested on:

    CPU / SoCIntel Core i5-6500T (4 cores, no GPU)
    RAM16 GB
    OSDebian 12
    Model testedQwen3 8B at Q4_K_M
    Result8B weights ~4.9 GB fit 16 GB with headroom; ~5 tokens/s generation on CPU

    Last tested: 21 September 2026

    What’s next?

    The natural next steps from this guide:

  • How Much RAM Does a Local LLM Need?

    How Much RAM Does a Local LLM Need?

    The most common reason a local AI setup disappoints is a RAM mismatch: the model does not fit, or it fits so tightly that the machine swaps and everything grinds to a halt. The good news is that this is arithmetic, not guesswork. A model’s memory need is dominated by the size of its weights, and that size is printed right on the file. This article gives you the formula, and then proves it with three real models measured on one modest box — the same machine, the same day, the same software — so the numbers you see are directly comparable.

    Intermediate · 9 min · Local AI

    The formula: weights plus a working margin

    When a model loads, its weights have to sit in memory. For a GGUF file at 4-bit (the Q4_K_M quantization that most people use as a default), the weights-in-memory are almost exactly the file size on disk. On top of that the engine needs a working margin for the context window and the compute buffers. A reliable rule of thumb:

    Free RAM needed ≈ model file size + 1 to 2 GB.

    That is the whole trick. You do not need to understand attention heads or KV-cache math to size your machine. You look at the file, add a couple of gigabytes, and compare it to what you have free. If it fits with room to spare, it will run. If it does not, it will run badly or not at all.

    There is one nuance worth stating plainly: this is about RAM, not VRAM. On a CPU-only box the weights live in system memory and the CPU computes. If you have a GPU with enough VRAM, the weights can move there and generation gets much faster, but the “does it fit” question is the same — just answered against your VRAM instead of your RAM.

    What we actually measured

    We ran three Qwen3 models at 4-bit on a single box — a 4-core Intel i5-6500T with 16 GB of RAM, no discrete GPU — and recorded both the memory the weights took and the real speed. The benchmark was llama.cpp with four threads, measuring prompt processing (how fast it reads your input) and token generation (how fast it writes the answer).

    Model (Q4_K_M)Weights in RAMPrompt speedGeneration speedFree RAM left (of 16 GB)
    Qwen3 0.6B~0.5 GB~48 tokens/s~47 tokens/s~15 GB
    Qwen3 4B~2.3 GB~30 tokens/s~9 tokens/s~13 GB
    Qwen3 8B~4.9 GB~9 tokens/s~5 tokens/s~11 GB

    Two things jump out. First, an 8B model fits comfortably in a 16 GB box. Its weights are about 5 GB, so with the working margin you are using roughly 6–7 GB out of 16, leaving plenty of room for the OS, your other containers, and a healthy context window. The answer to “can a 16 GB mini PC run an 8B model?” is a clear yes — it is not even close to the edge.

    Second, and this is the part most articles skip: fitting and feeling fast are different things. On a CPU box the generation speed drops as the model grows. The 0.6B model streams back almost instantly. The 4B model is comfortable for everyday use. The 8B model produces a token about every 0.2 seconds, which for a long answer is a noticeable wait — it works, it is private, it is yours, but it is not instant. If your box only has a CPU, that trade-off between how smart the model is and how patiently you are willing to wait is the real decision.

    Turning this into a decision

    Match the model to the machine you actually have, not the one you wish you had:

    • 8 GB of RAM: run the small end — a 0.6B to a 4B model at 4-bit. It will feel fast and leave room for everything else. Do not push an 8B model on an 8 GB box; you will spend your session watching it swap.
    • 16 GB of RAM: the sweet spot for CPU local AI. A 4B model is the everyday workhorse (fast and useful), and an 8B model is the step up when you want more quality and can tolerate the slower generation.
    • 32 GB or more, or a GPU: you can move to 13B–14B models at 4-bit, or run an 8B model with a large context window and still feel fine. A GPU with 8 GB+ of VRAM is the single biggest speed upgrade, because it lifts the generation rate that a CPU is capped on.

    The quantization is doing real work in these numbers. The 4-bit Q4_K_M files are the sweet spot for most people: a good balance of quality and size. Going to a lower quantization (3-bit) shrinks the model and makes it fit more easily at a cost to quality; going to 6- or 8-bit improves quality but roughly doubles the memory, which is when a 16 GB box stops being comfortable. The GGUF and quantization guide covers what the names like Q4_K_M actually mean and when it is worth changing them.

    Context is the hidden second consumer

    The weights are the big, predictable number. The context window is the smaller, variable one: the more conversation history and reference text you keep in front of the model, the more extra memory it needs. A model that fits with a 4,000-token context may not fit the same model with a 32,000-token context on the same 16 GB box. If your setup fits at short context and starts failing or slowing on long ones, that is the context growing into your free RAM, not the model. If you need long context on a CPU box, keep the model in the smaller size class rather than fighting the memory.

    Troubleshooting: when it does not fit

    A few of the things you will actually see, and what they mean:

    “Out of memory” on load, or the model fails to start. The weights plus margin exceed free RAM. Drop the model size a class, or lower the quantization. Check how much is actually free with free -h — the OS and other containers count against it.

    It loads but the machine is unusable and the fan is screaming. That is swap. The model barely fit, so every inference paged memory to disk. It is not a speed problem, it is a fit problem. Same fix: smaller model or more RAM.

    It is much slower than the numbers above. Make sure you are using as many threads as physical cores (not logical), that nothing else is hogging the CPU, and that you are not running an 8-bit quantization by accident on a CPU box. The speed you should expect on a given model is the one in the table for your size class.

    Bottom line

    You do not need to guess whether your hardware can run a local model. Weigh the file, add a couple of gigabytes, and compare it to your free RAM: 8 GB runs the small models, 16 GB runs a comfortable 4B and a workable 8B, and a GPU is what makes it feel instant. The numbers in the table are measured, not estimated, and the method transfers to any model and any box — including the mini PCs that are the natural home for this kind of setup, covered in the home server hardware guide.

    Tested on:

    CPU / SoCIntel Core i5-6500T (4 cores, no GPU)
    RAM16 GB
    OSDebian 12
    Enginellama.cpp build dd1ea5243, 4 threads
    ModelsQwen3 0.6B / 4B / 8B at Q4_K_M
    Result8B weights ~4.9 GB fit 16 GB; 0.6B ~47 t/s, 4B ~9 t/s, 8B ~5 t/s

    Last tested: 21 September 2026

    What’s next?

    The natural next steps from this guide:

  • llama.cpp vs Ollama vs LM Studio: Which to Use

    llama.cpp vs Ollama vs LM Studio: Which to Use

    llama.cpp, Ollama and LM Studio all do the same fundamental job — load a GGUF model and generate text — and they all use the same model files, so choosing between them is not a question of capability. It is a question of how you want to work: how much control you want, how little setup you tolerate, and whether you are on a desktop or a headless server. This guide puts the three side by side, using real measurements from the lab box (a 4-core Intel i5-6500T, 16 GB RAM, Debian 12, no discrete GPU) wherever we could measure, and a clear recommendation for each situation. If you only read one thing: on a CPU box where speed matters, llama.cpp is the fastest; on a server where you want it to “just work” and be scriptable, Ollama is the best default; on a desktop where you want the friendliest experience, LM Studio wins.

    Intermediate · 11 min · Comparison

    The same engine, three front ends

    It is worth stating plainly that these are not three competing technologies. All three load GGUF model files and produce tokens. Ollama and LM Studio both build on the same inference foundation that llama.cpp provides, and they differ in packaging. That means:

    • The model files are interchangeable. A GGUF you use in one runs in the others. You are not choosing a model ecosystem, only a way to drive it.
    • The differences are in control, convenience and interface. llama.cpp exposes everything as flags. Ollama hides it behind a CLI and an API. LM Studio hides it behind a graphical window.
    • Speed differences are real but secondary. The engine is shared, so the gaps are in overhead and defaults, not in some deep architectural advantage. On a CPU box the direct llama.cpp path is the most efficient; the wrappers are close behind and the difference rarely matters compared to model choice.

    llama.cpp: control and pure-CPU performance

    llama.cpp is the bare engine you build and drive yourself. You build it once from source, then run a model with explicit flags for context, threads and host. What you get in return is the most direct path to the hardware and the best pure-CPU numbers. On the lab box we benchmarked a 4.65B model at Q4_K with four threads and measured about 48 tokens per second processing a 512-token prompt and about 14 tokens per second generating. Those are honest CPU figures for a modest box, and they are what you get when nothing sits between you and the engine.

    The cost is that there is no hand-holding: you find the model file yourself, you manage it, you set every parameter by hand, and the “server” is a process you start with a command. This is the right choice when you want to tune context length and threads, when you are on a CPU box where every token per second counts, or when you want to understand exactly what is running. It is the least friendly option, and the most capable one.

    Ollama: the low-maintenance default

    Ollama wraps the same engine in a single install and a tiny CLI, and adds a model registry that removes the “find the right file” step. ollama pull gets you a verified model; ollama run talks to it; and an HTTP API on port 11434 (plus an OpenAI-compatible endpoint) means every chat app and script can use it without custom code. On the lab box we pulled a 0.6B model at Q4_K_M (a 522 MB file, verified on download) and measured token generation at roughly 49 to 52 tokens per second and prompt processing at roughly 140 to 235 tokens per second. For a model that small those are “instant” numbers, and the point is less the raw speed than how little stood between us and a working answer.

    This is the best default for a home server: it installs cleanly, it is scriptable, it is what most third-party tools assume, and it stays out of your way. The trade-offs are that the registry is a curated list (a very specific self-downloaded file is imported rather than pulled by name), and it is a little less tunable than raw llama.cpp. For most people on a server, that trade is the right one.

    LM Studio: the friendliest desktop experience

    LM Studio is a graphical desktop app for macOS, Windows and Linux. You browse a model catalogue, download with a click, chat in a window, and adjust parameters with sliders. It is the fastest way to “have local AI on my computer today” with zero terminal, and the parameter sliders make experimenting with model behavior genuinely easy. It also serves the loaded model on port 1234 as an OpenAI-compatible API, so it can do the server job too. We have not screenshotted the GUI in this series because the lab box is headless, so treat the desktop specifics as the documented behavior; the parts that matter on a server — the port 1234 server and the OpenAI-compatible API — are the documented and verifiable ones.

    Where LM Studio gives ground is that it is a GUI app, so it is the wrong shape for an always-on headless server, and its management layer adds a little overhead compared to a bare command-line runtime. It is the best on-ramp and the best place to discover which models you like, and the natural next step from it is to hand the same GGUF file to a headless runtime for production. There is no lock-in, because the model format is shared.

    Side by side

    Summarizing what matters in a decision:

    • Setup. llama.cpp: build from source, manage files yourself. Ollama: one install script, one pull command. LM Studio: install an app, click to download. From most to least effort: llama.cpp, Ollama, LM Studio.
    • Interface. llama.cpp: terminal and flags. Ollama: terminal CLI plus HTTP API. LM Studio: graphical window plus optional HTTP server.
    • Model management. llama.cpp: you source and store the files. Ollama: curated registry plus import. LM Studio: in-app catalogue that stores models for you.
    • Control and tuning. llama.cpp: full (every flag is yours). Ollama: good (API and some options). LM Studio: good via sliders, less under the hood.
    • Best on CPU speed. llama.cpp is the most direct and measured the cleanest CPU numbers in the lab; the others are close.
    • Best for a headless server. Ollama, for the combination of easy install, scriptability and a standard API. llama.cpp if you want maximum control.
    • Best for a desktop / first time. LM Studio.

    Which one should you pick

    Match the tool to the situation rather than picking a “best” in the abstract:

    • You have a CPU box and you want the most performance with your hands on the controls: llama.cpp. Build it, load your model, set threads to your core count, and benchmark. It is the most capable and the least hand-holding.
    • You have a home server and you want a local model that is easy to install, easy to script, and friendly to the tools you already use: Ollama. It is the default for a reason, and it is what a chat front end like Open WebUI expects.
    • You are on a desktop and you want to try local AI now, with a window and no terminal: LM Studio. It is the friendliest, and it is the best place to learn which models and quantizations you actually prefer before committing to a server setup.

    And because the model files are the same GGUF across all three, you can start with the friendliest one and move to the more capable one without losing anything. A common path is to learn on LM Studio, then run the model you settled on with Ollama on the server, and reach for llama.cpp only when you need to tune or squeeze. The decision is about how you want to work, and the model you run is independent of that choice.

    Common mistakes

    Picking based on benchmarks you read rather than your own hardware. Published speed numbers depend on CPU, RAM, GPU and threads in ways that do not transfer. Run the benchmark on your box with the model you care about — llama.cpp’s benchmark tool does this directly, and Ollama reports timing in its API — and decide on your own numbers.

    Thinking you are locking yourself into a model ecosystem. You are not. The GGUF file is the constant; the runtime is swappable. If a tool stops fitting how you work, move the same model file to another one.

    Over-engineering the first setup. The fastest path to a useful local model is the least-friction tool you will actually keep using. Start with the one that matches your comfort (often Ollama on a server, LM Studio on a desktop), get one real answer back, and only then optimize toward llama.cpp if the speed or control genuinely requires it.

    How this fits the rest of your home server

    Whatever you pick here, the rest of the local AI stack is identical. The model is a GGUF file chosen by the size-and-quantization logic, the front end is a chat UI like Open WebUI, the security story is “keep the port private and reach it over your VPN” (the Tailscale guide), and the box you are running it on is the one you chose in the hardware guide. The runtime is the one swappable layer in the whole thing, and because it sits on top of a shared model format, the choice is a preference, not a commitment. Pick the front end that matches how you work, run a model you measured on your own hardware, and put a chat window in front of it — that is the whole stack, and the runtime is the only part you get to change your mind about freely.

    Tested on:

    OSDebian 12
    Hardware4-core / 16 GB, no GPU
    llama.cpp (4.65B Q4_K, 4 threads)48.4 t/s prompt / 13.6 t/s generation
    Ollama v0.34.0 (qwen3:0.6b Q4_K_M)~49-52 t/s generation

    Last tested: 15 September 2026

    What’s next?

    The natural next steps from this guide:

  • LM Studio: The Easiest Way to Run Local LLMs

    LM Studio: The Easiest Way to Run Local LLMs

    LM Studio is the friendliest on-ramp to local AI: a desktop app where you browse a catalogue of models, download one with a click, and start chatting in a window — no terminal, no config file, no “what is a context window” moment. It is the option to choose when you are working on a machine with a screen and you want the path of least resistance. Under the hood it runs the same kind of GGUF models that llama.cpp and Ollama use, and it can also switch on a local server so the rest of your tools can talk to it over the network. One honest note up front: this guide is written from the app’s documented behavior on macOS, Windows and Linux. The lab box for this series is a headless server without a desktop, so we have not screenshotted the GUI here; the steps and the ports and the server behavior are the documented ones, and the parts we can verify on a headless box — the server port, the OpenAI-compatible API — are what you will actually lean on if you run it on a server.

    Beginner · 9 min · Desktop app

    What LM Studio is and who it is for

    LM Studio is a graphical desktop application, available for macOS (including Apple Silicon), Windows and Linux. Its three jobs are:

    • Find and download models. It includes a search interface over a catalogue of GGUF models, so you can look at size, quantization and author, and download what fits your machine without hunting around a model host.
    • Chat with them. Once a model is downloaded, you pick it, set a few generation parameters with sliders, and talk to it in a chat window that behaves like a normal chat app.
    • Serve them. A “Local Server” toggle exposes the loaded model on http://localhost:1234 as an OpenAI-compatible API, so any client written for the OpenAI API can use it by changing the base URL. This is the part that turns a desktop app into a small model server.

    The model files it uses are the same GGUF files the other two runtimes load, so a model you download in LM Studio can be loaded in llama.cpp or Ollama too. There is no lock-in; LM Studio is a choice of interface, not a closed format.

    Installing it

    Download the installer for your platform from the LM Studio site and run it. On macOS and Windows it is a standard installer; on Linux it ships as an AppImage (or a platform package), which you make executable and run. After install, launch the app and you are in the interface. There is no separate “server daemon” to install on the desktop — the server is a feature you switch on inside the running app (see the server section), which is the thing to keep in mind if you are on a headless box: the desktop app expects a display, so on a server without a GUI the headless path is the one you want.

    Downloading your first model

    Inside the app, use the search/browse view to find a model. The two numbers to check are the parameter count and the quantization, exactly as in the size-and-quantization guide: pick something your RAM can hold. As a safe first download on a 16 GB machine, a 4B to 8B model at 4-bit is the same sweet spot recommended everywhere in this series. Select it and download. LM Studio stores models in a local models folder it manages for you, so you do not have to decide where the file lives or name it — a convenience, at the cost of less control over the exact file, which is the flip side of the friendliness.

    Chatting and tuning

    Select the downloaded model, and the chat window is ready. Above the input there are a handful of generation parameters — temperature, the context length, the maximum response length — exposed as sliders rather than flags. This is the main appeal for a first model: you can change how the model behaves and see the effect immediately, without editing a command. A good habit is to keep temperature modest (around 0.2 to 0.7) for factual or coding tasks and raise it only when you want more variety. The context slider is the one with a real cost: a longer context uses more memory, so if the model feels slow or the app warns about memory, that is the first knob to turn down.

    Turning it into a server on port 1234

    This is the step that makes LM Studio useful beyond the machine it is installed on. In the app, open the Local Server area and enable the server. It listens on http://localhost:1234 and exposes the loaded model through an OpenAI-compatible API. After that, any existing OpenAI client points at LM Studio by changing the base URL to that address and setting any API key value (the local server does not require a real key, but the clients expect the field to be filled). The endpoints line up with the ones you would use on the OpenAI API — a model list and a chat-completions endpoint — so the change is a configuration edit, not a rewrite. Two practical notes. First, the server reflects the model currently loaded in the app, so switching models in the UI changes what the server serves. Second, the server runs while the app runs; on a desktop that is fine, but it is also why the desktop app is the wrong shape for an always-on headless server, where you would want a service that starts at boot and has no window.

    Running it headless on a server

    If your target is a headless box (like most home servers), the desktop app is not the right vehicle, because it is built around a graphical window. There are two honest ways to run an LM-Studio-style setup headless. One is a community “headless server” package that runs the model server as a background service without the GUI and manages the model for you. The other — and the one we would actually recommend for a server — is to skip the GUI entirely and use a runtime that is native to the command line: Ollama or llama.cpp. The model files are the same GGUF files, so nothing is lost by moving from the desktop app to a headless runtime; you keep the models and change the front end. This is worth saying plainly: LM Studio shines on a desktop, and for a 24/7 server the command-line runtimes are the better fit, even though they all speak the same model format.

    Keeping it private

    The server defaults to localhost, which is the safe default: only the machine it runs on can reach it. If you want a chat app or another device on your network to use it, you can point the server at your LAN address, but the cleaner pattern is to keep the server on the box and reach it through your VPN — the Tailscale in Docker guide shows how to get a device on your home network from anywhere without opening a public port. And the same rule as the rest of this series: a local model server has no account system, so its security is entirely “who can reach the port”. Keep that on loopback or a private network, and if you do want a named, HTTPS front for the UI, the Caddy in Docker guide is the way to add it. The security guide has the firewall and network-isolation details.

    Common gotchas

    The server says the model is not available. The server serves whatever model is loaded in the app. If you downloaded a model but did not select it, or you switched models after starting the server, the API will not see the one you expect. Load the intended model, then start (or restart) the server.

    Port 1234 is already in use. If something else is bound to 1234, the server will not start or will collide. Change the port in the server settings, or free the port. This also bites when a desktop instance with the server on and a headless daemon are both running at once — two processes, one port. Pick one owner for the port.

    It is slower than Ollama or llama.cpp on the same model. The GUI and its management layer add overhead, and the desktop app is not optimized the way a bare command-line runtime is. For a desktop that is usually a non-issue; for a server where speed matters, the headless runtimes are the better tool.

    You expect the desktop app to run on a headless box. It is a GUI app. On a server with no display it will not do its job, which is exactly why the headless section above points you to a service or a command-line runtime instead.

    How this fits the rest of your home server

    LM Studio is the “I have a computer with a screen and I want to play with local AI today” option, and it is the best one for that job. It is also the place to start if you are deciding which models you actually like, because the one-click downloads and instant parameter changes make experimentation cheap. Once you know which model and which quantization you want to run for real — and you will, after a little playing — the natural move for a home server is a headless runtime that serves that same GGUF file around the clock: Ollama for the low-maintenance path, llama.cpp for the control-and-performance path. And whatever you run, the next step is the same: a chat front end the whole house can use, covered in the Open WebUI guide, and a bit of monitoring so the loaded model does not quietly take the RAM your other services need. The models are the constant across all of it; only the front end changes.

    What’s next?

    The natural next steps from this guide:

  • Ollama: Install and Run a Local LLM in 5 Minutes

    Ollama: Install and Run a Local LLM in 5 Minutes

    Ollama is the fastest way from “I have a server” to “I have a working local model”. One install script, one command to pull a model from its registry, one command to run it — and you get a model you can talk to from the terminal, plus an HTTP API that every chat app and script can use. It wraps the same inference engine that powers the rest of the local AI world, so it is not a toy or a separate technology; it is that engine with a friendly front end. This guide installs it, pulls a model, runs it, measures it, and covers the one behavior that trips people up on the newer thinking models. Everything was done on the lab box: a 4-core Intel i5-6500T, 16 GB of RAM, Debian 12, no discrete GPU.

    Beginner · 10 min · CLI

    What Ollama actually is

    Ollama is a small program with three parts working together:

    • A server that stays running in the background, holds the loaded models in memory, and answers requests. You usually start it once and forget it.
    • A CLI (ollama) that talks to that server. ollama pull, ollama run, ollama list, ollama ps. This is what you type.
    • An HTTP API on localhost:11434 with both its own native endpoints and an OpenAI-compatible one, so existing OpenAI clients work with no changes.

    The model registry is the part that makes it feel effortless. Instead of hunting for a file and checking it matches your hardware, ollama pull qwen3:0.6b downloads the right file and verifies it. The trade-off for that convenience is that the registry is a curated list, so if you want a very specific model file you downloaded yourself, you import it rather than pull it by name.

    Installing it

    On the lab box we installed a specific release (v0.34.0) by downloading the binary distribution and running the server directly, which is the same shape as the official install but without needing root for the whole thing. The official one-liner does this for you:

    curl -fsSL https://ollama.com/install.sh | sh

    If you are running as a normal user without the rights the installer wants (as we were in the lab), download the release tarball for your platform, extract it into a folder of your own, and run the server binary directly:

    OLLAMA_HOST=127.0.0.1:11434 ./bin/ollama serve

    The OLLAMA_HOST variable is the one worth knowing. By default the server listens on 127.0.0.1, which is exactly what you want on a private server — nothing outside the machine can reach it. Set it to 0.0.0.0 only if you deliberately want the rest of your network to reach the API, and then you should be putting it behind your VPN rather than exposing it. We confirmed our server was answering {"version":"0.34.0"} on its version endpoint before doing anything else.

    Pulling and running a model

    Once the server is up, pulling a model is one command. In the lab we pulled a small, fast model to prove the pipeline:

    ollama pull qwen3:0.6b

    This downloaded a 522 MB model (a 0.6B-parameter model at Q4_K_M) and verified its checksum — the pull finished in under a minute on the lab connection. After it completes, ollama list shows it with its size, and you can run it interactively:

    ollama run qwen3:0.6b "What is the capital of France?"

    That is the whole “I have a local model” moment: type a question in the terminal, get an answer from a model running on your own hardware. For scripting, the same model is reachable over the API:

    curl http://localhost:11434/api/generate -d '{
  "model": "qwen3:0.6b",
  "prompt": "Say hello",
  "stream": false
}'

    And for any client written against the OpenAI API, the /v1/chat/completions endpoint behaves the same way, which is what makes Ollama the default assumption for third-party chat apps.

    Measuring it

    Ollama reports timing in the API response, so you get the real numbers without a separate benchmark tool. On the lab box, a chat request against the 0.6B model came back with the response text, the tokens generated, and the timing for both reading the prompt and generating the answer. What we measured:

    • Token generation: roughly 49 to 52 tokens per second. That is the answer-writing speed, and at that rate a normal-length reply streams back in well under a second. For a 0.6B model on a 4-core CPU with no GPU, that is comfortably in “instant” territory.
    • Prompt processing: roughly 140 to 235 tokens per second depending on the run, which is the speed at which it reads your input back. In practice that is fast enough that you will not notice it for normal prompts.

    These are small-model numbers, and that is the point of starting small: you get an immediate, pleasant experience that proves the whole setup works, before you spend time on a bigger model that is slower to load and slower to answer. When you move to a larger model, watch the generation rate in the response and decide against that baseline whether the extra intelligence is worth the extra wait.

    The thinking-model gotcha

    This is the one that cost us real debugging time in the lab, and it is worth knowing before it confuses you. A number of newer models — the Qwen3 family is a good example — are “thinking” models: before they produce the final answer, they generate an internal chain of reasoning. Ollama returns that reasoning in its own field, separate from the final answer. The practical consequence is that if you read only the main answer field and that field is short or empty, it can look like the model failed to respond at all, when in fact it did its work in the reasoning field and then gave a one-word answer.

    In the lab, a request to a thinking model returned its reasoning (a few hundred characters of “let me work this out…”) in the separate field and a short final answer in the main field. Two things follow from this. First, it is a model behavior, not an Ollama bug — the server did exactly what the model told it to. Second, if you are writing a client, read the full response, not just the primary content field, or you will occasionally see “empty” answers that are actually complete. And because thinking burns tokens on reasoning, a thinking model with a tight token limit can spend its whole budget thinking and return little or no final answer — raise the limit if you see that pattern.

    Keeping it running and keeping it private

    On a server you want Ollama running as a service so it survives a reboot, and you want its model store on a volume that survives a reinstall. In the lab the model store lived in a dedicated folder, and the whole thing fit on the same NVMe drive as the rest of the stack — a model or two is a few gigabytes, not a storage event. For the “I want it as a container” setup, which is the most reproducible way to run it alongside a chat UI, the official Ollama image does exactly this: a container for the runtime, a volume for the models, and the API on the standard port. That two-container pattern (Ollama plus a chat front end) is what we ran end to end in the lab, and it is covered in the Open WebUI guide. For reaching the UI from your phone or laptop without opening a port, the Tailscale in Docker guide is the clean path, and the security guide covers why the API should stay on loopback or a private network by default.

    Common gotchas

    The CLI says it cannot connect to a running Ollama. The server and the CLI are two separate processes, and the CLI only works while the server is running and on the host it expects. If you started the server with a custom OLLAMA_HOST, the CLI needs to know the same address. In the lab this showed up whenever the background server had been restarted without the host variable set. Check the server is actually up first; it is the more common cause than a configuration mismatch.

    A pull is slow or stalls. Model files are large, and the registry is a public service, so pull speed is whatever the internet is giving you. A stalled pull is usually a network hiccup, and re-running the same pull resumes rather than starting over. If you are on a slow line, a smaller quantization is the real fix, not patience.

    The model is in the list but the first request is very slow. The first request after a model has been idle loads its weights into memory, which takes a moment; subsequent requests are fast. This is normal, not a fault. If every request is slow, the model is too big for the box and it is swapping — see the size-and-quantization guide for the memory math.

    Your script gets an empty answer from a thinking model. Covered above, but it is the most likely thing to look like a bug when it is not. Read the full response including the reasoning field, and raise the token limit if the model is burning its budget on reasoning.

    How this fits the rest of your home server

    Ollama is the “set it and forget it” option in the local AI stack: the least fiddly install, the easiest model management, and an API that the rest of your tools already understand. It is the right default for most home servers, and the right thing to put a chat interface in front of. If you want to squeeze more out of a CPU box or you want your hands on the engine, the llama.cpp guide is the deeper route; if you are working on a desktop with a screen, LM Studio is the friendlier one. Whichever runtime you pick, the natural next step is a proper chat window that the whole house can use — the Open WebUI guide — and a bit of monitoring so a loaded model does not quietly take RAM from the services it shares the box with.

    Tested on:

    OSDebian 12
    Hardware4-core / 16 GB
    Ollamav0.34.0
    Modelqwen3:0.6b (Q4_K_M, 522 MB, verified on pull)
    Measured speed~49-52 t/s generation / ~140-235 t/s prompt

    Last tested: 15 September 2026

    What’s next?

    The natural next steps from this guide:

  • llama.cpp: Build, Run and Benchmark a Local LLM (CPU)

    llama.cpp: Build, Run and Benchmark a Local LLM (CPU)

    llama.cpp is the engine under the hood of most local AI setups, and it is also the most direct way to run a model when you have a CPU box and want to get the most out of it. It is a C++ program: you build it once from source (or grab a prebuilt binary), point it at a model file, and it generates text — from the command line, or as a small HTTP server that other apps can talk to. There is no account, no registry to sign up for, and nothing that phones home. This guide builds it, loads a real model, measures how fast it actually runs, and shows the exact flags that matter. Everything was done on a 4-core Intel i5-6500T with 16 GB of RAM and no discrete GPU, so the numbers here are what a modest CPU box really delivers.

    Intermediate · 12 min · C++ / CLI

    What llama.cpp gives you

    When you build llama.cpp you get a family of small tools in a single build/bin folder. The ones you will actually use are:

    • llama-server — runs a model as an HTTP server with an OpenAI-compatible API. This is what you point a chat app or a script at. It is the one we used in the lab.
    • llama-cli — an interactive chat in your terminal. Good for a quick “does this model work” check.
    • llama-bench — measures prompt-processing and token-generation speed for a model. This is how we got the numbers below, and it is the tool to reach for whenever you want to compare models or quantizations objectively.
    • llama-quantize — converts a full-precision model to a smaller quantized version (Q4, Q5, Q8, …). Useful when you have a full model and want to shrink it to fit your memory.

    The reason people use it directly rather than a wrapper is control: you choose the context length, the number of threads, the GPU split, and you see exactly what is happening. The cost is that there is no one-click “pull this model” — you manage the model files yourself.

    Building it

    The build needs a C++ compiler and CMake. On a Debian/Ubuntu box:

    sudo apt update
sudo apt install -y build-essential cmake git

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)

    On the lab box this completed cleanly with the default CPU backend. The key point for a CPU-only machine: leave GPU support off (which is the default when you have no CUDA toolkit installed). If you do have an NVIDIA GPU, you install the CUDA toolkit and add -DGGML_CUDA=ON to the cmake -B build line, and the same build will offload the model weights to the GPU. We built a CPU Release build here, which is the configuration that produces the honest CPU numbers in this guide.

    After the build, everything you need is in build/bin. Confirm it with ./build/bin/llama-cli --version. In the lab we were on a recent build (commit dd1ea5243), which is what produced the binary that ran the tests below.

    Loading and running a model

    A model for llama.cpp is a .gguf file — one file, self-contained, with the weights and the chat template inside it. You get these from a model host such as Hugging Face (the GGUF versions), and you put them anywhere on disk. You do not need a specific folder structure; the tools just take a path.

    For a quick interactive test, llama-cli is the fastest way to see if a model works at all:

    ./build/bin/llama-cli -m /path/to/model-Q4_K_M.gguf -c 4096 -t 4

    The flags that matter:

    • -m — the model file. Everything else hangs off this.
    • -c — the context window in tokens. This is how much of the conversation the model can “see” at once. It costs memory, so do not set it larger than you need. 4096 is a fine default; we used 32768 for the server run below.
    • -t — the number of CPU threads. Match this to your physical cores. On the 4-core lab box, -t 4 is the honest choice; setting it to 8 would just add contention.

    Running it as a server

    For real use, run llama-server so your apps can talk to it over HTTP. In the lab this is exactly the command we used:

    ./build/bin/llama-server \
  -m /home/chikewa/models/gemma-4-E2B-it-UD-Q4_K_XL.gguf \
  --host 0.0.0.0 --port 8080 \
  -c 32768 \
  --api-key-file /home/chikewa/api-keys.txt

    This serves an OpenAI-compatible API on port 8080. Once it is up, the model list lives at /v1/models and chat completions at /v1/chat/completions, so any client written for the OpenAI API works unchanged. We confirmed the server answered {"status":"ok"} on its health endpoint and returned the loaded model on /v1/models before sending any real requests. Two notes from the lab: bind --host 0.0.0.0 only if you genuinely intend the rest of the network to reach it; on a server that should stay private, --host 127.0.0.1 is the safer default and you reach it through your VPN. And the --api-key-file flag means the server will require a key on requests, which is worth turning on the moment anything other than your own machine might see the port.

    Measuring how fast it actually is

    This is the part that turns a guess into a decision. llama-bench runs a prompt-processing test (how fast it reads your input) and a token-generation test (how fast it produces the answer), and prints tokens per second for each. On the lab box, with the 4.65B-parameter gemma model at Q4_K and -t 4:

    ./build/bin/llama-bench \
  -m /home/chikewa/models/gemma-4-E2B-it-UD-Q4_K_XL.gguf \
  -p 512 -n 128 -t 4

    The result we measured:

    • 512-token prompt processing: about 48.4 tokens per second. This is the “it reads your long prompt” speed. At ~48 t/s, a 1,000-token prompt is digested in roughly 20 seconds.
    • 128-token generation: about 13.6 tokens per second. This is the “it writes the answer” speed — the number you feel. 13.6 t/s means a 200-word answer streams back in around 30 seconds, which is genuinely usable for everyday tasks even though it is not instant.

    These two numbers are what you want to capture for your own model and your own hardware, because they are the ground truth for whether a bigger model is worth the memory. Run llama-bench on the model you are about to commit to, and if generation drops below a pace you find pleasant, the fix is usually a smaller model or a higher quantization trade-off — or a GPU, which is the single biggest jump available.

    Common gotchas

    It is slower than the benchmarks claim. The most common cause is the context length. -c is not free: a 32k context holds a lot of state in memory and can slow generation noticeably compared to a 4k context, especially on CPU. If a model is “supposed” to be fast but is not, halve the context and re-benchmark before blaming the model. The second cause is thread count set above your physical cores, which adds contention rather than speed.

    The model does not fit, or the box swaps. If you load a model bigger than your available RAM, the operating system will start swapping to disk and the speed will collapse from tens of tokens per second to a trickle. Watch memory while it loads; if free RAM goes negative and swap climbs, the model is too big for the box at that quantization. The fix is a smaller model or a lower quant, per the size-and-quantization guide.

    The server answers but my client gets an empty response. With a few newer “thinking” models, the model spends its first tokens on internal reasoning that is returned in a separate field rather than in the main answer. If your client only reads the primary content field, it can look like the model returned nothing even though it worked. This is a model-behavior detail, not a llama.cpp bug — check the full response for a separate reasoning field before assuming the server is broken. We hit this with a thinking model in the lab, which is why it is worth knowing.

    Two versions of the binaries, or a stale build. If you rebuilt and the old behavior persists, you are almost certainly running the old binary from a different path. which llama-server and a fresh --version tell you exactly which build you are executing.

    How this fits the rest of your home server

    llama.cpp is the “I want the model itself, with my hands on it” option in the local AI stack. It is the right first choice on a CPU box because it is the most efficient of the three runtimes on that hardware, and it is the one to reach for when a wrapper is hiding something you want to change. The model files it loads are the same GGUF files that Ollama and LM Studio use, so nothing here is a dead end — if you later want the one-click model downloads or a nicer desktop, you move the runtime and keep the models. For the “just give me a model” path, see the Ollama guide; for the desktop route, the LM Studio guide; and for putting a proper chat window in front of whatever you run, the Open WebUI guide. And because a loaded model will happily hold several gigabytes of RAM, it belongs on a box you are watching — the Prometheus setup is the natural companion, so a chatty model does not quietly crowd out the rest of your services.

    Tested on:

    OSDebian 12
    HardwareIntel i5-6500T, 4 cores / 16 GB, no GPU
    llama.cppbuild dd1ea5243 (b10343-12), CPU Release build
    Model4.65B-param model at Q4_K (2.95 GB)
    Measured speed48.4 t/s prompt (512) / 13.6 t/s generation (128), 4 threads

    Last tested: 15 September 2026

    What’s next?

    The natural next steps from this guide:

  • GGUF and Quantization Explained: Pick the Right Model Size

    GGUF and Quantization Explained: Pick the Right Model Size

    Every local model you run is described by a name like some-model-8B-Q4_K_M.gguf, and almost everyone treats those bits after the name as magic strings. This guide takes them apart. What is a GGUF file, what do the two numbers in a model name actually mean, what a quantization level really costs in memory and quality, and how to pick one for the box you have. It is the reference the rest of the series points to, because once you understand size and quantization, “which model should I run” stops being a guess and becomes arithmetic.

    Beginner · 10 min · Concepts

    What a GGUF file is

    GGUF is a single-file format for storing a model. Inside one .gguf file you get the model’s weights, the architecture description, the chat template (the instructions that tell the model how to format a conversation), and the vocabulary. That single-file property is the whole point: there is no folder of parts to assemble, no separate tokenizer to match, nothing to forget. You copy one file from machine to machine, or download one file, and it runs. llama.cpp, Ollama and LM Studio all read it, which is why the model ecosystem is built on it. A model you download as GGUF for one of those tools is the same file that the others load — the format is the common currency, and the tools are just different front ends for it.

    The two numbers in a model name

    Take a name like model-8B-Q4_K_M. The 8B is the parameter count: roughly eight billion parameters. A parameter is a number the model learned, and the count is a coarse measure of how much it can hold in its “head”. More parameters generally means more capability, but also more memory and slower generation, all else equal. The Q4_K_M is the quantization, and it is the second, equally important half of the name. You will see the parameter count written as 0.6B, 1.5B, 4B, 8B, 14B, 32B, 70B and so on. As a rule of thumb for the “B” number: the smaller ones are fast and lightweight, the ones in the 4B to 14B range are the current sweet spot for home hardware, and the 32B and 70B models are the high end that want a real GPU or a lot of RAM.

    What quantization means

    A model’s parameters are originally stored at high precision, typically 16-bit or 32-bit numbers. Storing and computing on those is accurate but heavy. Quantization is the process of storing each parameter with fewer bits, trading a little precision for a lot less memory and (often) faster computation. The “Q” in the name stands for the number of bits per parameter: Q8 is 8 bits, Q4 is 4 bits, and so on. Going from 16 bits down to 4 bits shrinks the file to a quarter of its size. The cost is that each parameter is now an approximation of the original, and the model’s output quality degrades a little as you compress harder. Quantization is what makes running a 70B model at all possible on consumer hardware — at full precision it would need hundreds of gigabytes, but quantized it can fit in a fraction of that.

    Reading the quantization codes

    The codes you see in practice, from highest to lowest quality, are roughly:

    • F16 / F32 — full precision. The original, uncompressed values. Used for training and as a starting point; far too big to run locally.
    • Q8_0 — 8-bit. Very close to full precision in quality, and about half the size. The “safe” high-quality option when your memory allows it.
    • Q6_K — 6-bit, a “K-quant”. A good quality/size compromise.
    • Q5_K_M — 5-bit. Often the best balance for models you want to run well without huge memory.
    • Q4_K_M — 4-bit. The most common default and the one to understand. The “_K” means it is a mixed-precision K-quant: some parts of the model are kept at slightly higher precision than others, which preserves more quality than a flat 4-bit would. The “_M” is a size variant of that scheme. Q4_K_M is the quantization most model hosts offer first, and for good reason: it is small enough to run on modest hardware and good enough that the quality loss is often barely noticeable.

    There are also lower options (Q3, Q2 and below) that squeeze the model much smaller at a real quality cost, and they are worth using when the alternative is not being able to run the model at all. The practical guidance is: reach for the highest quantization your memory can hold. If a model’s Q5 does not fit but its Q4 does, run the Q4. Do not run a Q4 when your box could hold the Q5 — the extra quality is free if you have the memory.

    The memory math, with real numbers

    Here is the arithmetic that turns a model name into a memory requirement. The weight size in memory is approximately the parameter count times the bits per parameter, divided by 8 to get bytes. Worked out for a few real cases on the kind of hardware in this series:

    • A 4.65B model at Q4 is about 2.95 GB of weights. That is the exact model we ran in the lab: it loaded into a 16 GB box with room to spare for a 32k context, and generated at about 14 tokens per second on a 4-core CPU. The file was 2.95 GB on disk, and the loaded model fit comfortably with several gigabytes of RAM left over for the rest of the system.
    • A 0.6B model at Q4_K_M is about 0.5 GB. This is the model we used for the Ollama tests; it loaded and generated almost instantly, which is why starting with a small model is the right first move.

    Two things to add to the weight size. First, the context window costs memory on top of the weights, and it grows with the context length, so a 32k context needs more RAM than a 4k one for the same model. Second, if you have a GPU, you want the weights to fit in VRAM, not system RAM — an 8 GB card can hold roughly an 8B model at Q4 plus a modest context, which is why “8B at 4-bit” is the number people aim for on an 8 GB GPU. The whole exercise is the same everywhere: count the parameters, apply the bits, add the context, and check it fits the memory you actually have. If it does not, drop the quantization or the model size, not your expectations about the rest of the box.

    How to pick a quantization for your box

    A method that avoids most dead ends:

    1. Find your memory budget. Free system RAM for a CPU box, or VRAM for a GPU. This is the ceiling.
    2. Reserve room for context and the OS. Do not spend the entire budget on weights. Leave a couple of gigabytes for the context and the rest of the system, or you will be swapping.
    3. Pick the model size first, then the highest quant that fits. Decide how smart you need it to be (the “B” number), then take the best quantization your remaining memory allows. Q8 if you can, else Q6, Q5, Q4 in that order.
    4. Prefer a smaller model at a higher quant over a bigger one at a lower one, up to a point. A 7B at Q5 often beats a 13B at Q3, because the lower quant degrades quality faster than the extra parameters add it. There is a crossover where the bigger model wins, but for home hardware the smaller-and-cleaner choice is usually right.
    5. Measure, do not assume. Run the model, note the tokens per second, and judge the output on a few real tasks. The numbers tell you whether the quantization you picked is the right trade for your use.

    Common mistakes

    Picking by parameter count only. “70B is the best, I want 70B” is how people end up with a model that cannot load on their box. The parameter count and the quantization are one decision, not two. Always do the memory math before you commit to the size.

    Running a low quant when a higher one fits. If your box can hold the Q5, running the Q4 is leaving quality on the table for free. Check the memory, then take the best quant that fits.

    Setting a huge context on a small box. The context is real memory, and it is the quiet thing that pushes a “should fit” model into swapping. If a model loads but is mysteriously slow, the context length is the first thing to cut.

    Mixing a model and quantization from different sources without checking they match. The GGUF file already has its quantization baked in, so a mismatch is not really possible within a single file — but it is worth knowing that the quantization is a property of the file you download, not a setting you apply later. You cannot “quantize down” a file you already have without running a quantization tool on the full-precision weights.

    How this fits the rest of your home server

    Size and quantization are the vocabulary that connects every other guide in this series. When the llama.cpp guide tells you to check whether a model fits, this is the math it means. When the Ollama guide says to start with a small model, this is why 0.5 GB loads instantly and 2.95 GB is the sweet spot on a 16 GB box. When you are deciding whether a bigger model is worth it, the tokens-per-second number you measure is the answer to the question this guide sets up. The model file is the constant: pick the size and quantization that fit your memory, download it once as a GGUF, and any of the runtimes can load it. That is the entire decision, reduced to arithmetic and one measurement.

    Tested on:

    Hardware4-core / 16 GB
    4.65B model at Q4_K2.95 GB on disk, runs in 16 GB RAM with a 32k context
    0.6B model at Q4_K_M522 MB on disk, loads instantly

    Last tested: 15 September 2026

    What’s next?

    The natural next steps from this guide:

  • Run Local AI on Your Home Server: The Complete Guide

    Run Local AI on Your Home Server: The Complete Guide

    Local AI has crossed a line: you no longer need a data-centre’s worth of silicon to run a model that answers questions, drafts text and summarizes documents. On the same mini PC or old desktop you already use for your home server, you can run a large language model entirely on your own hardware — no API key, no per-token bill, no prompt ever leaving your network. This guide is the map for that whole territory. It tells you what your machine can realistically run, which of the three main runtimes (llama.cpp, Ollama and LM Studio) fits how you work, how to pick a model by its size and quantization, and how to put a proper chat interface in front of it. Every command and number below was checked on real lab hardware, and each section links out to the dedicated guide for that piece of the stack.

    Beginner · 12 min · Local AI

    Why people bother: a local model is private by default, works offline, costs nothing after the hardware, and you can swap models whenever you like. The trade-off is that the quality ceiling is set by your RAM and GPU, so the first honest question is not “which app” but “what can my box actually run”.

    What your hardware needs to run

    The single most important number is how much of a model’s weights fit in memory at once. A model’s on-disk size is a close proxy for the memory it will want when loaded, plus a margin for the context window you give it. As a working rule:

    • Roughly 4–8 GB of free memory runs small models (around 0.5B to 4B parameters at 4-bit) comfortably on CPU. These are fast and good enough for summarization, quick Q&A and drafts.
    • 8–16 GB opens up mid-size models (7B–14B at 4-bit). On CPU these are slower but usable for short tasks.
    • A GPU with 8+ GB of VRAM changes everything: it lets you load the model weights into fast video memory and generate at tens of tokens per second instead of a handful. This is the difference between “it answers” and “it feels instant”.

    In the lab we used a 4-core Intel i5-6500T with 16 GB of RAM and no discrete GPU. It ran a 4.65B-parameter model at 4-bit at about 48 tokens per second reading a prompt and 14 tokens per second generating. That is genuinely usable for everyday tasks — you type, and the answer streams back in a couple of seconds. The point is not that this box is powerful; it is that a modest box is enough to start, and you can always grow into a GPU later.

    If you are deciding what to buy specifically for this, the home server hardware guide has real benchmarks and the same “mini PC is the best all-rounder” conclusion applies: a used mini PC with 16 GB of RAM is the cheapest way into local AI that does not feel like a toy.

    The three runtimes, and when to use each

    Under the hood, all three popular tools are doing the same job: loading a GGUF model file and serving it. They differ in how much they hide and how much control they give you.

    • llama.cpp is the foundation. It is a C++ program you build once, and it gives you the most control and the best pure-CPU performance. You drive it from the command line. Choose it when you want maximum performance on a CPU box, you are comfortable in a terminal, or you want to tinker with context length, threads and quantization yourself.
    • Ollama wraps the same engine in a single install script and a tiny CLI plus an API. One command pulls a model from a registry; another runs it. It is the fastest way to “just have a local model” and the one most tools assume. Choose it for a low-maintenance, scriptable setup.
    • LM Studio is a desktop app with a graphical interface: browse a model catalogue, one-click download, chat in a window, and flip on a local server when an app needs the API. Choose it if you want the friendliest on-ramp and you are working on a machine with a screen rather than a headless server.

    The good news is that the model files are interchangeable across all three. A GGUF you download for Ollama can be loaded in llama.cpp, and vice versa. So pick the runtime for how you like to work, not out of fear of being locked in. A practical comparison of the three, with the numbers we measured, is in the llama.cpp vs Ollama vs LM Studio comparison.

    Choosing a model: size and quantization

    Every model you will run is described by two numbers: its parameter count (the “B” number — 0.6B, 4B, 8B, 14B, 70B) and its quantization (Q4_K_M, Q5_K_M, Q8_0, and so on). More parameters generally means smarter output; a higher quantization means less quality lost when the model is compressed to fit in memory. The two trade against each other on the same memory budget: a smaller model at a higher quant often outperforms a bigger model at a lower one, up to a point.

    As a starting point on a 16 GB box with no GPU, a 4B to 8B model at 4-bit is the sweet spot. It loads quickly, leaves room for a long context, and is fast enough to be pleasant. If you add an 8 GB GPU, an 8B to 14B model at 4- or 5-bit is the target. What each quantization level actually costs in memory and quality — including what “Q4_K_M” means rather than treating it as a magic string — is broken down in the GGUF and quantization explainer.

    Putting a chat interface in front of your model

    A raw API endpoint is powerful but unfriendly for day-to-day use. Most people want a chat window with conversations, a model picker and a place to paste documents. Open WebUI is the most popular self-hosted option: it runs in a Docker container, connects to Ollama (or any OpenAI-compatible endpoint) in a couple of lines of config, and gives you a clean, chat-app-style interface that works in a browser on any device on your network. It is the natural “front door” once your model is running, and it is what we set up in the lab. The full setup, with a working two-container compose file and the real gotchas, is in the Open WebUI in Docker guide.

    Keeping it safe on your network

    Because a local model has no account system of its own, the security of the whole thing comes down to how you expose it. The defaults are your friend: bind the model server and the web UI to 127.0.0.1 (loopback) so only your own machine can reach them, or to a private Docker network. To use the UI from your laptop or phone, put it behind your VPN rather than opening a port to the internet — Tailscale in Docker is the simplest way to reach a service on your home network from anywhere without punching holes in your router. If you do want the UI reachable across the house, the home server security guide covers firewalls and Docker network isolation, and Caddy in Docker shows how to front it with HTTPS if you need a proper domain. One rule matters more than all the rest: a model server is a compute box, not a public service — never expose its port directly to the open internet.

    A practical order to follow

    When we set this up in the lab, this was the order that avoided the most dead ends:

    1. Decide the budget. Check your free RAM and whether you have a GPU. This sets which model size you are realistic about.
    2. Pick the runtime. Terminal and control: llama.cpp. Low-maintenance and scriptable: Ollama. Desktop and graphical: LM Studio.
    3. Run one small model first. A 0.5B to 4B model at 4-bit. Get one real answer back before you spend time on anything bigger. This is the fastest way to prove the whole pipeline works.
    4. Measure it. Note tokens per second. Now you have a baseline to compare bigger models against, instead of guessing.
    5. Grow. Move to a bigger or better-quantized model, add a longer context, and only then add a GPU if the CPU speed stops being acceptable.
    6. Add the front door. Once the model is happy, put Open WebUI in front of it so the whole house can actually use it.

    Each of those steps is its own guide. Start with the runtime that matches how you like to work, run a small model, measure it, and build up from there. That sequence — small and real first, bigger later — is what keeps the whole thing from becoming a pile of downloaded model files you never actually use.

    How this fits the rest of your home server

    Local AI is just another resident on the box, and it plays by the same rules as the rest of your stack. It wants a fair share of RAM and, if you give it a GPU, the whole GPU while it is loaded; keep an eye on the box with the Prometheus and node_exporter setup so a chatty model does not quietly eat memory your media server needs. Back up the models folder and the runtime’s state with the same approach as the 3-2-1 backup strategy — a model is a few gigabytes of file, and re-downloading it is the only thing slower than losing it. And because the model lives on the server and the people using it live on their phones and laptops, the self-hosting starter guide is the right place to recap how all of this sits together on a machine you already run. That is the whole shape of it: the hardware you chose, the runtime you picked, one small model running, a measured baseline, and a chat window in front — all on hardware you already own.

    What’s next?

    The natural next steps from this guide: