Chikewa

Start here

Search

Author: Chikewa

  • Can a Mini PC Run a Local AI Model?

    Can a Mini PC Run a Local AI Model?

    A local LLM on a mini PC is one of the most attractive “what can this little box actually do” questions in homelabbing, because the answer used to be an obvious no. This article answers it with a real test: we ran an 8-billion-parameter model on the same modest hardware a typical 16 GB mini PC would have, and measured whether it fits and how it actually feels. No GPU, no cloud, no — just the box and the numbers.

    Intermediate · 8 min · Hardware

    What “run an LLM” requires, in one line

    The model’s weights have to fit in memory, and then the CPU (or a GPU, if you have one) has to compute. The weights are the predictable part — for a 4-bit model the file size is almost exactly the memory it takes. An 8B model at 4-bit is about 5 GB of weights, so it fits in a 16 GB box with room to spare for the operating system, your other containers, and a healthy context window. That is the short answer: yes, a 16 GB mini PC runs an 8B model.

    But “runs” and “feels good” are different, and on a CPU-only box that gap is the whole story. The more you want the model to be smart, the more it has to compute per token, and a small CPU has a hard ceiling on how many tokens it can produce per second.

    What we measured on a CPU-only box

    We tested on a 4-core Intel i5-6500T with 16 GB of RAM and no discrete GPU — deliberately an older, modest CPU, because that is the honest lower bound for “a cheap mini PC.” We ran three Qwen3 models at 4-bit and measured both memory and real speed:

    Model (Q4_K_M)Weights in RAMGeneration speedPractical feel
    Qwen3 0.6B~0.5 GB~47 tokens/sNearly instant
    Qwen3 4B~2.3 GB~9 tokens/sComfortable everyday use
    Qwen3 8B~4.9 GB~5 tokens/sUsable, but you wait on long answers

    Two conclusions. First, all three fit in 16 GB with room to spare, so memory is not what stops you from running a bigger model on a mini PC. Second, the 8B model is the boundary of “comfortable” on a CPU: it answers, it is genuinely useful for summarising and drafting and Q&A, but at about five tokens per second a long reply takes a noticeable while. The 4B model is where CPU-only local AI feels good on hardware like this.

    The deeper point is that a mini PC is the right shape of machine for this, even though it is not fast. It uses a little power, it is quiet, it runs the OS and your other Docker services in the same 16 GB, and the model lives in the headroom. You are not buying a GPU; you are buying a box that runs the whole server and a useful model in the margin.

    When the answer becomes a stronger yes

    Three things move a mini PC from “runs an 8B model” to “runs local AI that feels good”:

    • A faster CPU. A modern N100 or a mini PC with more, faster cores raises the token rate meaningfully, because CPU generation speed scales with the cores available. The 8B model that took ~0.2s per token on our 2015 four-core box will be noticeably snappier on current hardware.
    • A GPU. This is the big one. A discrete GPU with 8 GB+ of VRAM moves the weights into fast memory and lifts the generation rate to tens of tokens per second, which is the difference between “it answers” and “it feels instant.” If local AI is a primary use, a mini PC with a modest GPU is the step up.
    • Picking the right size. For CPU-only, a 4B model at 4-bit is the sweet spot — fast enough to feel responsive, smart enough to be useful. Reach for 8B when you need the quality and can accept the slower generation.

    The how-much-RAM-a-local-LLM-needs article in this cluster has the full measured table and the formula for sizing any model to any box, and the run local AI on your home server guide covers the runtimes (llama.cpp, Ollama, LM Studio) you would actually use on the mini PC.

    Power and cost: the quiet advantage

    One reason a mini PC is the natural home for a local model is the cost side. A small box idles at a fraction of what a GPU workstation draws, and local inference on CPU draws modestly more than idle. The total cost of a useful local model is the electricity of a box that was already running your media server and your other services — the model is an add-on, not a second machine. We do not have a validated power figure for this specific configuration to quote, so treat the exact wattage as something to measure on your own box rather than assume; the structural point (a model living in the headroom of an already-running box) holds regardless.

    Bottom line

    Can a mini PC run a local LLM? Yes — a 16 GB mini PC runs an 8B model comfortably in memory, and a 4B model runs it in a way that feels good on CPU. The honest caveat is speed: on a CPU-only box, the model’s intelligence and its responsiveness pull in opposite directions, and 4B is where the balance lands for most people. If you want an 8B model to feel instant, the upgrade is a faster CPU or a GPU, not more RAM. For the Chikewa reader who already runs a home server, the mini PC is the practical machine for local AI: it fits in the same 16 GB as everything else, it is quiet and cheap to run, and the numbers above tell you exactly what to expect before you buy it.

    Tested on:

    CPU / SoCIntel Core i5-6500T (4 cores, no GPU)
    RAM16 GB
    OSDebian 12
    Model testedQwen3 8B at Q4_K_M
    Result8B weights ~4.9 GB fit 16 GB with headroom; ~5 tokens/s generation on CPU

    Last tested: 21 September 2026

    What’s next?

    The natural next steps from this guide:

  • Immich vs Google Photos: Can You Actually Replace It?

    Immich vs Google Photos: Can You Actually Replace It?

    Google Photos is where most people’s pictures already live, and “what if I stopped using it” is one of the most common self-hosting motivations — usually a mix of privacy, control, and not wanting a cloud service to decide what happens to a decade of family photos. Immich is the leading self-hosted answer. This is an honest comparison of the two, including the things Immich still does not match, and what it actually takes to move off Google Photos rather than run the two side by side.

    Intermediate · 8 min · Immich

    What you get on each side

    Google Photos is a polished, finished product. Backup from your phone is automatic and invisible, search by text (“sunset at the beach”) and by object is excellent, the timeline is beautiful, and sharing is frictionless because your contacts are already there. The trade-offs are that it is a closed service, your data is on their infrastructure, it is subject to their terms and their pricing for extra storage, and the ML features and your photos are processed on their servers.

    Immich gives you the same core job — a photo library that backs up from your phone, groups by face, searches by object and by natural-language query, and stores the original files — but on your own hardware. The machine learning runs on your box, the originals are plain files in a folder you can copy anywhere, and there is no account and no service in the path. The setup is covered in the Immich Docker guide.

    Where Immich genuinely matches Google Photos

    • Automatic phone backup. The Android and iOS apps back up your camera roll to the server, and the phone keeps working offline. For the “stop losing photos if I lose my phone” job, it does it.
    • Face grouping. It detects and groups faces so you can find everyone, the way Google does.
    • Object and text search. The machine-learning side supports object detection and a natural-language query box, so “what did I shoot of the dog” works.
    • The originals are yours. Everything lands in a plain upload folder. Google Photos’ originals are accessible but tied to the service; Immich’s are literally files on your disk.

    Where Immich still does not match

    Being honest about the gap matters, because it is the difference between “replace it this weekend” and “run it alongside for a while.”

    • Polish of the web and timeline experience. Google Photos’ interface is more refined. Immich is good and improving, but a long session in the timeline and the editing tools is still a notch behind.
    • Sharing with people who are not on the system. Sharing a photo with a relative is frictionless in Google because they are there. In Immich, sharing with someone outside your server means sending them a file or a link, not a native share.
    • Search quality out of the box. Google’s search is trained on an enormous corpus and is very good immediately. Immich’s search depends on the ML models running on your hardware and takes time to index a large library.
    • Indexing time on first import. The ML side (faces, embeddings) is CPU-hungry on first import. A large library takes a while to fully index — hours on a decent desktop CPU, longer on a Raspberry Pi. The backup and browsing work from the first photo; the smart features catch up in the background.

    The migration is the real work

    Running Immich is straightforward; the hard part is getting your existing photos off Google Photos and into it, because Google does not offer a clean bulk download of the originals in a layout that Immich wants. The practical path is:

    1. Use Google Takeout to request your photos and videos as originals (not screenshots), which produces a large archive.
    2. Extract it and let Immich import the folder as a library — it scans the files and builds the timeline.
    3. Let the machine-learning indexing run in the background until faces and search catch up.

    After that, point your phone’s Immich app at the server for new photos, and you are running your own library with the old photos inside it. Plan for the Takeout export to be slow and large; it is the least glamorous step, but it is the only one that moves the history.

    Resource reality

    Immich is one of the heavier “simple” self-hosted apps, because it carries a real machine-learning pipeline. In the lab an empty-library Immich stack idled at roughly 1.3 GB of RAM across its four containers (server, database, machine-learning, and cache) on a 4-core box. A comfortable deployment wants at least 4 GB free for it, more if you are indexing a large library on CPU. That is real, but it is the same class of box that runs a media server, so for most homelabbers it fits.

    Whichever you pick

    If you…PickWhy
    Want zero effort, best-in-class search, and do not mind a closed cloudGoogle PhotosMost polished, best search, frictionless sharing
    Want your photos and the ML on your own hardware, no serviceImmichOriginals are files you own, no account, no cloud in the path
    Are on a home server already and want to reduce a dependencyImmichIt fits the same box and removes a recurring storage cost
    Mainly share photos with family who are all on GoogleGoogle PhotosNative sharing wins until your household is on Immich

    Bottom line

    Immich is a real replacement for the core job of Google Photos — automatic backup, face groups, object and text search, and originals you own — and for a home-server person it removes a recurring cloud dependency and keeps the data on hardware they control. The honest gaps are polish, out-of-the-box search quality, and the friction of sharing with people outside your system, plus a migration where the Google Takeout export is the slow part. If you are willing to do the one-time migration and accept that the smart features take time to index, Immich is the credible “I want to replace Google Photos” answer.

    Tested on:

    Immichv3.2.2 (4 containers)
    OS / runtimeDebian 12, Docker 29.7.2
    Idle RAM (empty library)~1.3 GB total (server 755 MiB, postgres 364 MiB, ML 169 MiB, cache 12 MiB)
    NoteA fresh install must use the current ghcr.io/immich-app/postgres image (see the Immich guide)

    Last tested: 21 September 2026

    What’s next?

    The natural next steps from this guide:

  • Tailscale vs Cloudflare Tunnel for Your Home Server

    Tailscale vs Cloudflare Tunnel for Your Home Server

    Both Tailscale and Cloudflare Tunnel solve the same problem — reaching your home server from outside without opening an inbound port on your router — but they solve it in opposite directions, and that difference decides which one fits your setup. This is a straight comparison of the two, across the things that matter (what connects to what, what it costs, what it exposes, and which clients it supports), ending with which one to pick in which situation.

    Intermediate · 8 min · Networking

    The fundamental difference: who connects to whom

    Tailscale builds a private mesh. Your server and your phone both run the Tailscale agent/app, sign in with the same account, and then they can talk to each other directly over an encrypted connection. There is no third party in the middle of your traffic (only an encrypted relay is used when two devices cannot reach each other directly). Your data goes from your phone to your server.

    Cloudflare Tunnel is a tunnel. Your server keeps a persistent outbound connection to Cloudflare’s edge, and your browser or client reaches the server by going to a Cloudflare domain, through that connection. Your traffic transits Cloudflare’s edge on the way to your server.

    Both require no open inbound ports, which is the whole safety win over port forwarding. The difference is that the mesh is peer-to-peer and private by construction, while the tunnel routes through a vendor’s edge.

    Cost

    Tailscale has a free tier that is genuinely usable for a personal home lab — a small number of users and nodes — and paid plans if you need more. For a household of a few devices, the free tier is usually enough.

    Cloudflare Tunnel (called Cloudflare Access / the tunnel feature) is free for personal use tied to a domain you control. You need a domain pointed at Cloudflare, which you likely already have. So both are effectively free for a home setup; the difference is not price, it is the architecture and what it takes to run them.

    Clients and where each one fits

    This is where the choice usually lands.

    • Tailscale needs the app on every device you want to reach the server from. That is trivial for phones and laptops, and great when you want a private network that also lets you reach any service on the box — not just one. It is the better fit when you want general remote access to your homelab, SSH included.
    • Cloudflare Tunnel needs no app on the client. Any browser can reach the service through the domain, which makes it the better fit when the client cannot install the Tailscale app — a browser on a work computer, some smart TVs, a tablet without the app. You get a real HTTPS URL, which is convenient for services you also want on a clean domain.

    In practice the two are often paired rather than opposed: Tailscale for the devices that can run it (the fast, private path), and a tunnel for the clients that cannot (a clean HTTPS domain). They are not mutually exclusive.

    Security and exposure

    Both keep your inbound ports closed, which is the main risk removed. The nuance is what is in the path of your traffic. With Tailscale the traffic is between your own devices, encrypted, with no vendor seeing it. With Cloudflare Tunnel the traffic transits Cloudflare’s edge; that is a well-established, reputable operator, but it is a third party in the path, and the service is reachable on a public domain (gated by whatever access rules you configure).

    If “no third party touches this” is a hard requirement, that is a point for Tailscale. If a stable public HTTPS domain is more important to you than keeping a vendor out of the path, that is a point for the tunnel. Both are a large step up in safety from a plain forwarded port, which is the one you should avoid unless a specific client forces it.

    Setup and day-to-day

    Tailscale is the simpler mental model: install the agent, sign in, done. It handles the networking for you, including reaching the box over SSH, which a tunnel does not give you by default. The setup is covered in the Tailscale in Docker guide, including the one step that trips people up (approving the login in the console).

    Cloudflare Tunnel requires a domain and a little more wiring (the tunnel process, a service definition, and ideally a reverse proxy in front so the same domain fronts more than one service). It is covered in the Cloudflare Tunnel in Docker guide, and pairs naturally with a proxy such as the one in the Caddy guide if you want one domain for several services.

    Which one to pick

    If you…PickWhy
    Want the simplest, most private remote access to your whole box (including SSH)TailscalePrivate mesh, no vendor in the path, app-based
    Need to reach one service from a client that cannot install an appCloudflare TunnelWorks from any browser via a clean HTTPS domain
    Want a stable public domain for a few self-hosted servicesCloudflare TunnelReal HTTPS URL, no inbound ports
    Want to do bothBothTailscale for the fast private path, tunnel for app-less clients — they complement each other

    Bottom line

    Neither is wrong, and they are not really competitors so much as two tools for two shapes of access. Tailscale is the default for a home-server person: it is private, it needs no vendor in the path, it is the simplest to reason about, and it reaches the whole box including SSH. Cloudflare Tunnel is the tool to reach a service from a client that cannot install an app, or when you want a clean public HTTPS domain. The common, sensible answer is to run Tailscale for your own devices and add a tunnel for the few clients that need a plain URL — both with your inbound ports closed, which is the real win over the old port-forwarding approach.

    What’s next?

    The natural next steps from this guide:

  • Jellyfin vs Plex: Which One Fits Your Home Server?

    Jellyfin vs Plex: Which One Fits Your Home Server?

    The honest question is not “which is better” — it is “which fits how I watch and what I am willing to pay.” Jellyfin and Plex do almost the same job (turn a folder of media into a streamable library), but they make very different bets on price, on who controls the transcoding, and on how much you are expected to trust the vendor. This is a straight comparison across the things that actually matter, without a crown, so you can pick the one that fits your household.

    Beginner · 7 min · Media Server

    Price: the clearest difference

    Jellyfin is free, open source, and free forever. There is no premium tier, no monthly fee, and no feature held back behind a paywall. Everything in the server is available to everyone.

    Plex has a free tier, but the features people actually want — transcoding on the server, direct play to certain clients, and direct streaming to some devices — live behind Plex Pass, a paid subscription. If you run Plex the way most people want to, you are paying a recurring fee. For a home server you own, that is the single biggest practical difference, and for a lot of people it is the deciding one.

    Transcoding: who does the work

    This is the technical heart of the difference. Both apps can direct-play (send the file as-is, cheap on the CPU) and transcode (re-encode for a device that cannot play the source, expensive on the CPU).

    Jellyfin lets you use your own hardware encoder — Intel Quick Sync, NVIDIA, AMD — for the transcoding, and it is free to do so. The ceiling on how many 4K streams you can transcode at once is set by your hardware, and you can measure it. On a modest 4-core Intel box with a basic iGPU we measured three concurrent 4K-to-1080p transcodes holding real time before it started to slip — that is the number a dedicated 4K transcoding test on the same box confirms.

    Plex bundles a licensed encoder and handles the hardware acceleration for you, which is convenient — but the advanced transcoding and hardware-accelerated paths are part of the paid tier. The trade-off is that Plex’s client experience is generally smoother and more polished, at the cost of the subscription.

    Clients and ease of use

    Plex wins on the out-of-the-box experience. Its apps are polished, the setup wizard is friendly, and the metadata, artwork and matching are excellent with almost no configuration. If you want the least-friction path from “folder of movies” to “it just works on every screen,” Plex is the smoother ride.

    Jellyfin’s clients are good but a notch behind in polish. The browser player works everywhere, the mobile apps are solid, and most major smart TVs either have a native app or can run it in the browser. If you are willing to spend an evening configuring it, you get the full feature set for free. The setup is well documented in the Jellyfin Docker guide.

    Where each one fits

    If you…Reach forWhy
    Do not want to pay a subscription and you are comfortable configuringJellyfinFull features, free, and you control the hardware transcoding
    Want the smoothest setup and you do not mind a subscriptionPlexPolished clients, great metadata, minimal fiddling
    Have a capable iGPU or GPU and want to transcode 4KJellyfinFree hardware transcoding, and you can measure your ceiling
    Mostly watch on a smart TV and want it to just workEitherBoth work; Plex is slightly smoother, Jellyfin is free
    Want everything to stay on your own hardware, no vendor cloudJellyfinNo account, no service dependency, open source

    The privacy and control angle

    Both run the media server on your own machine, so your files stay with you in both cases. The difference is in the control plane. Jellyfin has no account and no vendor service in the path of your data — the server is yours entirely. Plex ties the premium experience to a Plex account and, for remote access, routes through its infrastructure. If “no third party touches this” is a hard requirement for you, that is a point for Jellyfin, independent of price.

    Bottom line

    Neither is wrong. Jellyfin is the choice when you want full features, free, on your own terms, and you are happy to spend a little time on setup — it is the natural fit for a home-server person who is already running Docker and does not mind configuring. Plex is the choice when you want the most polished, lowest-friction experience and are comfortable paying a subscription for it. For the Chikewa audience — people who run their own boxes and like to understand what they are running — Jellyfin is usually the better match, and the hardware is where the real capability lives, not the app.

    What’s next?

    The natural next steps from this guide:

  • Access Jellyfin from Outside Your Home Network

    Access Jellyfin from Outside Your Home Network

    A Jellyfin server that only works on your home network is half the job. The real point is watching from the phone on the train, from the laptop at a friend’s place, or from the TV when you are travelling. Getting there without turning your house into an open target comes down to choosing one of three access paths — and for most people the right answer is the one that opens no ports at all. This article walks through the three, with the real trade-offs, and tells you which one fits your setup.

    Intermediate · 8 min · Networking

    The three ways to reach it from outside

    There are really only three approaches, and they differ in one fundamental way: who connects to whom.

    • Port forwarding (the old way). You open a port on your router so the internet can find your server. It works, but it exposes a port to the whole world, which is why it needs a reverse proxy, HTTPS, and careful firewall rules to be safe.
    • A private mesh (Tailscale). Your phone and your server both join a private encrypted network. They talk to each other directly, and nothing is exposed to the public internet. This is the path most home-server setups should use.
    • A tunnel (Cloudflare Tunnel). Your server initiates an outbound connection to Cloudflare’s edge, and your browser reaches the server through that. Like the mesh, it needs no open inbound ports, but it routes traffic through Cloudflare rather than a direct peer-to-peer link.

    The mesh and the tunnel both avoid open inbound ports, which is the single most important safety property. Port forwarding is the one you reach for only when you have a specific reason to, such as needing a plain domain with a normal client that cannot install the mesh app.

    For a home server the Tailscale approach is the best fit, because it is the safest with the least moving parts. Install the agent on the server, install the app on your phone and laptop, sign in with the same account, and the devices see each other on a private network with their own addresses. You then point your Jellyfin client at the server’s private address and you are streaming, with every byte encrypted and with nothing on your router opened to the outside.

    Two things make this the default recommendation. First, it is outbound-only: your server initiates the connection, so there is no inbound port to secure. Second, it is private by construction — the traffic never transits a third-party network, it goes directly between your own devices (with an encrypted relay used only when two devices cannot reach each other directly). If you want a working, tested setup for this, the Tailscale in Docker guide has the exact compose file and the one step that trips people up (approving the login in the console).

    The honest trade-off: every device you want to stream from needs the Tailscale app. That is not a problem for a household of phones and laptops, but a smart TV that cannot install the app is out — for that you pair the mesh with one of the other methods, or you accept that the TV only works at home.

    The alternative that needs no app on the client: Cloudflare Tunnel

    When you need to reach the server from a device that cannot run the mesh app — a browser on a work computer, some TVs, a tablet without the app — a tunnel is the better fit. Your server keeps an outbound connection to Cloudflare, and you reach Jellyfin through a real domain over HTTPS. There is no open inbound port, the certificate is handled for you, and any device with a browser can connect.

    The trade-offs are that the traffic transits Cloudflare’s edge (fine for most, but it is a third party in the path), and you need a domain pointed at Cloudflare. If that fits your setup, the Cloudflare Tunnel in Docker guide walks through the tunnel and the compose wiring. It pairs naturally with a reverse proxy so the same domain also fronts your other services — see the Caddy reverse proxy guide for that layer.

    Port forwarding, and when you actually need it

    Port forwarding is still valid in a narrow set of cases: a client that only speaks a plain URL and cannot install the mesh app or use a tunnel, or a network where you specifically want direct access. If you go this route you are taking on the full security job yourself — a reverse proxy in front, a real domain, HTTPS, a strong password, and a firewall that allows only the one port. The server hardening guide covers the SSH, firewall and Docker network side of that. For a first-time media server, we would reach for the mesh or the tunnel first and only fall back to port forwarding if a specific client forces it.

    Whichever path, a few rules keep it sane

    These apply to all three methods and are worth doing once:

    • Bind the server to the internal interface, not to every interface, so the only way in is the path you chose.
    • Use HTTPS for any access that transits the internet. The mesh is already encrypted; the tunnel gives you HTTPS; port forwarding needs a proxy to add it.
    • Keep a strong account password and separate users per household member, because a remote-access URL is more valuable to a stranger than a local one.
    • Prefer direct play on the client you can, because remote buffering is often a bandwidth problem more than a server problem, and a path that lets the client play the file directly will feel far better than one that forces a transcode across your internet connection.

    Bottom line

    For most home servers the answer is the private mesh: no open ports, encrypted, no third party in the path, and a tested setup that takes an afternoon. Add a tunnel when you need to reach the server from a client that cannot run the mesh app. Keep port forwarding in your back pocket for the rare client that forces it. Whichever you pick, the goal is the same — the same library on the same terms whether you are at home or anywhere else — and the method to choose is the one that exposes the least of your network while still reaching the devices you actually watch on.

    What’s next?

    The natural next steps from this guide:

  • How Much RAM Does a Local LLM Need?

    How Much RAM Does a Local LLM Need?

    The most common reason a local AI setup disappoints is a RAM mismatch: the model does not fit, or it fits so tightly that the machine swaps and everything grinds to a halt. The good news is that this is arithmetic, not guesswork. A model’s memory need is dominated by the size of its weights, and that size is printed right on the file. This article gives you the formula, and then proves it with three real models measured on one modest box — the same machine, the same day, the same software — so the numbers you see are directly comparable.

    Intermediate · 9 min · Local AI

    The formula: weights plus a working margin

    When a model loads, its weights have to sit in memory. For a GGUF file at 4-bit (the Q4_K_M quantization that most people use as a default), the weights-in-memory are almost exactly the file size on disk. On top of that the engine needs a working margin for the context window and the compute buffers. A reliable rule of thumb:

    Free RAM needed ≈ model file size + 1 to 2 GB.

    That is the whole trick. You do not need to understand attention heads or KV-cache math to size your machine. You look at the file, add a couple of gigabytes, and compare it to what you have free. If it fits with room to spare, it will run. If it does not, it will run badly or not at all.

    There is one nuance worth stating plainly: this is about RAM, not VRAM. On a CPU-only box the weights live in system memory and the CPU computes. If you have a GPU with enough VRAM, the weights can move there and generation gets much faster, but the “does it fit” question is the same — just answered against your VRAM instead of your RAM.

    What we actually measured

    We ran three Qwen3 models at 4-bit on a single box — a 4-core Intel i5-6500T with 16 GB of RAM, no discrete GPU — and recorded both the memory the weights took and the real speed. The benchmark was llama.cpp with four threads, measuring prompt processing (how fast it reads your input) and token generation (how fast it writes the answer).

    Model (Q4_K_M)Weights in RAMPrompt speedGeneration speedFree RAM left (of 16 GB)
    Qwen3 0.6B~0.5 GB~48 tokens/s~47 tokens/s~15 GB
    Qwen3 4B~2.3 GB~30 tokens/s~9 tokens/s~13 GB
    Qwen3 8B~4.9 GB~9 tokens/s~5 tokens/s~11 GB

    Two things jump out. First, an 8B model fits comfortably in a 16 GB box. Its weights are about 5 GB, so with the working margin you are using roughly 6–7 GB out of 16, leaving plenty of room for the OS, your other containers, and a healthy context window. The answer to “can a 16 GB mini PC run an 8B model?” is a clear yes — it is not even close to the edge.

    Second, and this is the part most articles skip: fitting and feeling fast are different things. On a CPU box the generation speed drops as the model grows. The 0.6B model streams back almost instantly. The 4B model is comfortable for everyday use. The 8B model produces a token about every 0.2 seconds, which for a long answer is a noticeable wait — it works, it is private, it is yours, but it is not instant. If your box only has a CPU, that trade-off between how smart the model is and how patiently you are willing to wait is the real decision.

    Turning this into a decision

    Match the model to the machine you actually have, not the one you wish you had:

    • 8 GB of RAM: run the small end — a 0.6B to a 4B model at 4-bit. It will feel fast and leave room for everything else. Do not push an 8B model on an 8 GB box; you will spend your session watching it swap.
    • 16 GB of RAM: the sweet spot for CPU local AI. A 4B model is the everyday workhorse (fast and useful), and an 8B model is the step up when you want more quality and can tolerate the slower generation.
    • 32 GB or more, or a GPU: you can move to 13B–14B models at 4-bit, or run an 8B model with a large context window and still feel fine. A GPU with 8 GB+ of VRAM is the single biggest speed upgrade, because it lifts the generation rate that a CPU is capped on.

    The quantization is doing real work in these numbers. The 4-bit Q4_K_M files are the sweet spot for most people: a good balance of quality and size. Going to a lower quantization (3-bit) shrinks the model and makes it fit more easily at a cost to quality; going to 6- or 8-bit improves quality but roughly doubles the memory, which is when a 16 GB box stops being comfortable. The GGUF and quantization guide covers what the names like Q4_K_M actually mean and when it is worth changing them.

    Context is the hidden second consumer

    The weights are the big, predictable number. The context window is the smaller, variable one: the more conversation history and reference text you keep in front of the model, the more extra memory it needs. A model that fits with a 4,000-token context may not fit the same model with a 32,000-token context on the same 16 GB box. If your setup fits at short context and starts failing or slowing on long ones, that is the context growing into your free RAM, not the model. If you need long context on a CPU box, keep the model in the smaller size class rather than fighting the memory.

    Troubleshooting: when it does not fit

    A few of the things you will actually see, and what they mean:

    “Out of memory” on load, or the model fails to start. The weights plus margin exceed free RAM. Drop the model size a class, or lower the quantization. Check how much is actually free with free -h — the OS and other containers count against it.

    It loads but the machine is unusable and the fan is screaming. That is swap. The model barely fit, so every inference paged memory to disk. It is not a speed problem, it is a fit problem. Same fix: smaller model or more RAM.

    It is much slower than the numbers above. Make sure you are using as many threads as physical cores (not logical), that nothing else is hogging the CPU, and that you are not running an 8-bit quantization by accident on a CPU box. The speed you should expect on a given model is the one in the table for your size class.

    Bottom line

    You do not need to guess whether your hardware can run a local model. Weigh the file, add a couple of gigabytes, and compare it to your free RAM: 8 GB runs the small models, 16 GB runs a comfortable 4B and a workable 8B, and a GPU is what makes it feel instant. The numbers in the table are measured, not estimated, and the method transfers to any model and any box — including the mini PCs that are the natural home for this kind of setup, covered in the home server hardware guide.

    Tested on:

    CPU / SoCIntel Core i5-6500T (4 cores, no GPU)
    RAM16 GB
    OSDebian 12
    Enginellama.cpp build dd1ea5243, 4 threads
    ModelsQwen3 0.6B / 4B / 8B at Q4_K_M
    Result8B weights ~4.9 GB fit 16 GB; 0.6B ~47 t/s, 4B ~9 t/s, 8B ~5 t/s

    Last tested: 21 September 2026

    What’s next?

    The natural next steps from this guide:

  • How Many 4K Streams Can Your Home Server Transcode?

    How Many 4K Streams Can Your Home Server Transcode?

    The question that decides whether a home server is worth it: how many 4K streams can it actually transcode at once? Not how many the marketing page promises, and not what a YouTube comment claims, but what the hardware does when three people in the house all hit play at the same time. This article answers that with a real test. We took a modest box, generated a genuine 4K source file, and ran Jellyfin’s hardware transcode path (Intel Quick Sync) under load, counting exactly how many simultaneous 4K-to-1080p transcodes the machine keeps up with before the picture starts buffering. Everything below was measured on the same machine, on the same day.

    Intermediate · 9 min · Jellyfin

    Why transcoding is the real limit

    Most home servers do not transcode at all. If your TV or phone can play the file directly, Jellyfin sends the raw stream over and the CPU sits idle — that is called direct play, and it is cheap. The problem only appears when a device cannot play the source: a phone that will not decode 10-bit HEVC, a smart TV that only wants 8-bit H.264 at a certain bitrate, or a client on a slow connection that needs the video scaled down and re-encoded. That is when the server has to transcode, and that is where the hardware either keeps up or it does not.

    On a modern Intel CPU the transcode is handed off to the integrated graphics unit through a feature called Quick Sync (QSV). The idea is that the video encoder is a dedicated block of silicon, so re-encoding a stream should cost the general-purpose cores almost nothing. In practice the ceiling is set by a mix of the encoder, the scaling step, the memory bus, and how many streams are competing for all of it. That is the number we wanted to find.

    The test setup

    We used a box that is deliberately unglamorous, because most people are not buying a $1,500 workstation for a media server:

    • CPU: Intel Core i5-6500T — a 4-core Skylake from 2015
    • Graphics: Intel HD Graphics 530 (the iGPU with Quick Sync)
    • RAM: 16 GB
    • OS / runtime: Debian 12, Docker 29.7.2
    • Jellyfin: 10.11.11, with ffmpeg 7.1.4, transcode running with the /dev/dri device passed through

    The source was a 30-second 4K (3840×2160) H.264 8-bit file at about 20 Mbps — a realistic streaming bitrate. Each test transcoded it to 1080p H.264 at 8 Mbps, which is exactly the kind of downscale a phone or TV triggers. To simulate N simultaneous viewers we ran N independent transcodes of the same file at once and watched how long a 30-second clip actually took. If a 30-second clip takes 30 seconds or less, the machine is keeping up with real time. If it takes longer, the viewer is buffering.

    How many streams it kept up with

    Concurrent 4K→1080p streamsTime to transcode 30 sKept up with real time?Peak RAM
    111 sYes — 2.7× faster than real time~437 MiB
    221 sYes — 1.4× faster~881 MiB
    332 sBarely — 0.94× (just under real time)~1.33 GB
    442 sNo — falls behind (0.71×)~1.76 GB
    664 sNo — clearly behind (0.47×)~2.63 GB

    Read that table as a household. One or two people streaming 4K that needs transcoding: completely comfortable, the box has headroom. Three: it is at its limit, holding real time but with no margin — add a fourth viewer or a second 10-bit title and it starts to slip. Four or more simultaneous transcodes: the picture buffers. For a box this old and this cheap, three concurrent hardware transcodes is a genuinely good result.

    The RAM line is worth noticing too. Each additional transcode added roughly 0.4–0.5 GB, so even six streams used less than 3 GB of the 16 GB available. RAM was never the constraint here — the encoder throughput was. That matters if you are sizing a machine: for a media server, the question is almost always the CPU and iGPU, not the memory, unless you are also running a lot of other heavy services.

    The gotcha: not every QSV pipeline works

    There is one thing that would have wasted hours, so it is documented here. We first tried to force the entire pipeline onto the iGPU — hardware decode, hardware scale, hardware encode — the fully-accelerated path. On this Intel driver (the iHD video driver) that combination failed with a filter error: “Impossible to convert between the formats supported by the filter”. The fully-accelerated chain simply did not negotiate a valid pixel format.

    The combination that worked — and the one that produced the numbers above — was CPU decode, a standard software scale to 1080p, and the QSV encoder for the final H.264 step. That is essentially what Jellyfin’s default transcode profile does, and it is the practical recommendation: let the encoder ride the iGPU and do the scaling on the CPU. If you are chasing the all-on-GPU path on an Intel iGPU, expect to hit driver limits; the encoder-offload path is the reliable one.

    What this means for your hardware

    This box is a 2015 Skylake. Newer Intel iGPUs — the iGPU in an Intel N100 mini PC, or anything Alder Lake and later — have faster QSV encoders and more of them, so a modern mini PC will comfortably do more than three simultaneous 4K transcodes and will transcode 10-bit HEVC and AV1 sources that this older chip struggles with. The exact ceiling will be different on your hardware, but the shape of the test is the same and you can repeat it on your own box: generate a 4K source, run the transcode with your real device passthrough, and count how many concurrent streams hold real time. The method is the point, because it turns “my TV buffers sometimes” into a number you can act on.

    A few things that change the number in practice:

    • Source format. 8-bit H.264 sources are the cheap case. 10-bit HEVC or AV1 sources cost more to decode, so the ceiling drops.
    • Output resolution and bitrate. Transcoding to 720p is cheaper than to 1080p; a higher output bitrate costs more.
    • How much is direct play. The more of your household that direct-plays, the fewer transcodes you actually ever need, and the more margin you have left.

    Troubleshooting a server that buffers

    If your Jellyfin is buffering when you know it should be able to keep up, work down this list:

    Confirm it is transcoding at all. If it is direct-playing, the buffer is a network problem, not a CPU problem. The web client and the TV app show which mode is active. If it is transcoding and buffering, you have hit the ceiling above.

    Check the device is actually passed through. On a Linux box the container needs /dev/dri mapped in and the user running it in the right group. If the passthrough is missing, Jellyfin falls back to a slow software encoder and the ceiling drops to a fraction of what the table shows.

    Look at the transcode profile. A profile that outputs 10-bit or a very high bitrate is doing more work per stream than it needs to be. For phones and TVs, 8-bit H.264 at a moderate bitrate is almost always enough.

    Watch the real resource. If the CPU is pinned and RAM is low, you are CPU-bound and the only real fixes are fewer concurrent transcodes, a cheaper output profile, or a faster machine.

    Bottom line

    For a 4-core 2015 box with a basic Intel iGPU: three concurrent 4K-to-1080p transcodes is the ceiling, RAM is not the bottleneck, the encoder-offload (not all-on-GPU) path is the reliable one, and a modern N100-class mini PC will do meaningfully better. If you are deciding what to buy for a media server, the transcoding headroom is the spec that actually predicts whether the household gets buffering — and it is a number you can measure on any candidate machine with the method above.

    Tested on:

    CPU / SoCIntel Core i5-6500T (4 cores, 2015)
    GraphicsIntel HD Graphics 530 (Quick Sync)
    RAM16 GB
    OSDebian 12
    Docker29.7.2
    Jellyfin10.11.11 (ffmpeg 7.1.4)
    Result3 concurrent 4K→1080p transcodes at real time; 4+ fall behind

    Last tested: 21 September 2026

    What’s next?

    The natural next steps from this guide:

  • Open WebUI in Docker: A Chat UI for Local LLMs

    Open WebUI in Docker: A Chat UI for Local LLMs

    A model running on your server is powerful, but reaching it through a raw API endpoint is not how a household uses anything. Open WebUI is the most popular self-hosted front end for local models: it runs in a single Docker container, connects to Ollama (or any OpenAI-compatible endpoint) in a couple of lines of configuration, and gives you a clean chat interface with conversations, a model picker and document upload that works in a browser on any device on your network. This guide runs it the way we ran it in the lab — as a two-container stack (Ollama for the model, Open WebUI for the interface) — and walks through the real setup, the first login, connecting the model, and the gotchas that actually appear. Every step below was done and verified on the lab box: a 4-core Intel i5-6500T, 16 GB RAM, Debian 12, Docker 29.7, no discrete GPU.

    Beginner · 11 min · Docker

    What Open WebUI is

    Open WebUI is a web application, not a model runner. It does not load models itself; it connects to a model backend (Ollama, llama.cpp’s server, or any OpenAI-compatible API) and puts a proper interface in front of it. What it adds over a raw API:

    • A chat interface that behaves like a normal chat app: conversations you can go back to, a model selector, streaming responses.
    • Multi-user support. Each person gets an account and their own conversation history. This is what turns “the model on the server” into “the family can each use it”.
    • Document and knowledge features so you can attach a file and ask questions about it, plus prompts, tools and other extensions.
    • A single container that keeps all of its state (users, chats, settings) in one folder, so the whole thing is trivial to back up and move.

    Because it is a front end, the rule for this guide is the same as for the rest of the stack: the model is provided by Ollama (or another runtime), and Open WebUI’s job is to make that model pleasant to use. We used Ollama as the backend in the lab, which is the most common and best-supported pairing.

    The two-container stack

    The cleanest, most reproducible setup is a compose file with two services on a shared Docker network: Ollama running the model, and Open WebUI running the interface, with the UI told to reach Ollama by its service name. This is exactly what we ran end to end in the lab, and a chat request went from the browser-facing container, through to the Ollama container, and back with a real model response.

    services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    volumes:
      - ./ollama:/root/.ollama
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "127.0.0.1:3005:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_SECRET_KEY=change-me-to-a-long-random-string
    depends_on:
      - ollama
    volumes:
      - ./open-webui:/app/backend/data
    restart: unless-stopped

    The details that matter. The OLLAMA_BASE_URL points at Ollama by its service name on the shared network (http://ollama:11434), which is why both containers need to be in the same compose project — they can resolve each other by name. The port is published as 127.0.0.1:3005, bound to loopback, so only the server itself can reach the UI directly; that is the safe default and you reach it from your devices over your VPN. The two volumes are the entire state: ./ollama holds the downloaded models and ./open-webui holds the users, chats and settings. Back up those two folders and you can rebuild both containers and lose nothing. The WEBUI_SECRET_KEY signs the app’s sessions — set it to a long random string rather than the placeholder, because it is what makes your logged-in state secure. If you have an NVIDIA GPU, add the standard GPU resource reservation to the Ollama service and the model will run on the GPU instead of the CPU.

    First boot and the admin account

    Run docker compose up -d. Both containers start; Open WebUI takes a little while on first boot while it initializes, so wait for it to report healthy (or for the page to stop showing its loading screen) before trying to log in. The very first thing you do when the UI opens is create the admin account — the first account registered becomes the administrator of the whole instance. Do this before anything else and write the credentials down, because there is no separate “first user” flow and losing them means resetting the auth in the state folder. In the lab we created the admin account and confirmed it by signing back in, which is the step worth doing before you trust anything else.

    Confirm the backend connection next. With the OLLAMA_BASE_URL set in the compose file, Open WebUI should already see the models that Ollama has. If the model list is empty, the UI cannot reach Ollama — which is almost always a network-name or port issue between the two containers, not a problem with either app. Once the model appears, the pairing is working.

    Getting a model in and having a real conversation

    If Ollama has no models yet, pull one. In the lab we used a small, fast model to keep the test clean: a 0.6B model at Q4_K_M, a 522 MB file that Ollama downloaded and verified. With it loaded, we sent a real chat request through the full path — from Open WebUI’s API, to the Ollama container, and back. The response came back with a proper model answer, which is the end-to-end proof that the stack works: the interface, the backend, and the model are all talking to each other correctly. In the UI the same thing is just “type a message and read the answer”, but the API path is worth confirming once, because it is the thing a chat front end is for, and seeing a real response is the fastest way to know the whole chain is healthy.

    One behavior to expect with the newer thinking models: some spend their first tokens on internal reasoning that the backend returns in a separate field, and the final answer can be short. If a reply looks unexpectedly brief, that is usually the model’s doing, not a broken connection. And because the UI is where a whole household will live, set a sensible default model and a reasonable context length once, so every user gets a sane experience without fiddling.

    Security: the part that matters most

    Open WebUI has accounts, but the model backend it fronts does not, and the security of the whole setup is “who can reach the ports”. Keep the UI on loopback (as the compose file does) and reach it from your devices through your VPN — the Tailscale in Docker guide is the clean way to get your phone or laptop onto your home network from anywhere without opening a public port. Do not publish the UI port to your LAN or the internet just because it is convenient; a model front end on an open port is an open door to your server’s compute and to whatever the backend can do. If you do want a named, HTTPS front, the Caddy in Docker guide shows how to add one in front of the loopback port. The security guide covers the firewall rules and network isolation, and setting a real WEBUI_SECRET_KEY (not the placeholder) is the small thing that keeps the app’s own session handling secure.

    Common gotchas

    The model list is empty. The UI cannot reach the backend. In the two-container setup this is almost always that the two services are not on the same network, or the OLLAMA_BASE_URL does not match the service name and port. Check that both containers are up, that they share the compose project, and that the URL points at the right name and port. This was the single most common thing in the lab, and it is a configuration issue, not a broken app.

    The page loads but you cannot do anything / you are logged out repeatedly. Usually the state volume is not persisting, so the app restarts its users on every reload. Confirm the ./open-webui volume is actually being written to and survives a container restart — a fresh data folder on each start is the tell.

    It is slower than the model “should” be. Open WebUI adds a little overhead, but the dominant factor is still the model and the hardware. If the backend (Ollama or llama.cpp) is slow, the UI will be slow; measure the backend directly before blaming the front end. On the lab’s CPU box, small models felt instant and larger ones felt slower, which is the model and the hardware, not the interface.

    First boot looks hung. The container does real work on first start (initializing its database and configuration) and the page shows a loading screen while it does. Give it a moment before concluding it failed; check the container logs if it does not come up, which will say what it is waiting on.

    How this fits the rest of your home server

    Open WebUI is the front door of the local AI stack, and it is the step that turns a working model into something the household actually uses. It pairs with whatever runtime you chose — Ollama in this guide, but the same pattern works with llama.cpp’s server or any OpenAI-compatible backend — and it sits on the same security rules as everything else: loopback by default, reached over your VPN, optionally fronted by HTTPS. Its state is one folder, so it belongs in your normal backup routine alongside the model store, and because a loaded model holds real RAM, it is another reason to have the box watched by monitoring. The shape of the finished setup is the whole series in one picture: the hardware you chose, a model you measured and sized for that hardware, a runtime that serves it, and this chat window in front — all private, all on hardware you own, and all reachable from the devices in the house over a network you control.

    Tested on:

    OSDebian 12
    Docker29.7.2
    Hardware4-core / 16 GB
    StackOpen WebUI (container) + Ollama (container), shared network
    Modelqwen3:0.6b Q4_K_M (522 MB)
    Verifiedend-to-end chat through the full two-container stack

    Last tested: 15 September 2026

    What’s next?

    The natural next steps from this guide:

  • llama.cpp vs Ollama vs LM Studio: Which to Use

    llama.cpp vs Ollama vs LM Studio: Which to Use

    llama.cpp, Ollama and LM Studio all do the same fundamental job — load a GGUF model and generate text — and they all use the same model files, so choosing between them is not a question of capability. It is a question of how you want to work: how much control you want, how little setup you tolerate, and whether you are on a desktop or a headless server. This guide puts the three side by side, using real measurements from the lab box (a 4-core Intel i5-6500T, 16 GB RAM, Debian 12, no discrete GPU) wherever we could measure, and a clear recommendation for each situation. If you only read one thing: on a CPU box where speed matters, llama.cpp is the fastest; on a server where you want it to “just work” and be scriptable, Ollama is the best default; on a desktop where you want the friendliest experience, LM Studio wins.

    Intermediate · 11 min · Comparison

    The same engine, three front ends

    It is worth stating plainly that these are not three competing technologies. All three load GGUF model files and produce tokens. Ollama and LM Studio both build on the same inference foundation that llama.cpp provides, and they differ in packaging. That means:

    • The model files are interchangeable. A GGUF you use in one runs in the others. You are not choosing a model ecosystem, only a way to drive it.
    • The differences are in control, convenience and interface. llama.cpp exposes everything as flags. Ollama hides it behind a CLI and an API. LM Studio hides it behind a graphical window.
    • Speed differences are real but secondary. The engine is shared, so the gaps are in overhead and defaults, not in some deep architectural advantage. On a CPU box the direct llama.cpp path is the most efficient; the wrappers are close behind and the difference rarely matters compared to model choice.

    llama.cpp: control and pure-CPU performance

    llama.cpp is the bare engine you build and drive yourself. You build it once from source, then run a model with explicit flags for context, threads and host. What you get in return is the most direct path to the hardware and the best pure-CPU numbers. On the lab box we benchmarked a 4.65B model at Q4_K with four threads and measured about 48 tokens per second processing a 512-token prompt and about 14 tokens per second generating. Those are honest CPU figures for a modest box, and they are what you get when nothing sits between you and the engine.

    The cost is that there is no hand-holding: you find the model file yourself, you manage it, you set every parameter by hand, and the “server” is a process you start with a command. This is the right choice when you want to tune context length and threads, when you are on a CPU box where every token per second counts, or when you want to understand exactly what is running. It is the least friendly option, and the most capable one.

    Ollama: the low-maintenance default

    Ollama wraps the same engine in a single install and a tiny CLI, and adds a model registry that removes the “find the right file” step. ollama pull gets you a verified model; ollama run talks to it; and an HTTP API on port 11434 (plus an OpenAI-compatible endpoint) means every chat app and script can use it without custom code. On the lab box we pulled a 0.6B model at Q4_K_M (a 522 MB file, verified on download) and measured token generation at roughly 49 to 52 tokens per second and prompt processing at roughly 140 to 235 tokens per second. For a model that small those are “instant” numbers, and the point is less the raw speed than how little stood between us and a working answer.

    This is the best default for a home server: it installs cleanly, it is scriptable, it is what most third-party tools assume, and it stays out of your way. The trade-offs are that the registry is a curated list (a very specific self-downloaded file is imported rather than pulled by name), and it is a little less tunable than raw llama.cpp. For most people on a server, that trade is the right one.

    LM Studio: the friendliest desktop experience

    LM Studio is a graphical desktop app for macOS, Windows and Linux. You browse a model catalogue, download with a click, chat in a window, and adjust parameters with sliders. It is the fastest way to “have local AI on my computer today” with zero terminal, and the parameter sliders make experimenting with model behavior genuinely easy. It also serves the loaded model on port 1234 as an OpenAI-compatible API, so it can do the server job too. We have not screenshotted the GUI in this series because the lab box is headless, so treat the desktop specifics as the documented behavior; the parts that matter on a server — the port 1234 server and the OpenAI-compatible API — are the documented and verifiable ones.

    Where LM Studio gives ground is that it is a GUI app, so it is the wrong shape for an always-on headless server, and its management layer adds a little overhead compared to a bare command-line runtime. It is the best on-ramp and the best place to discover which models you like, and the natural next step from it is to hand the same GGUF file to a headless runtime for production. There is no lock-in, because the model format is shared.

    Side by side

    Summarizing what matters in a decision:

    • Setup. llama.cpp: build from source, manage files yourself. Ollama: one install script, one pull command. LM Studio: install an app, click to download. From most to least effort: llama.cpp, Ollama, LM Studio.
    • Interface. llama.cpp: terminal and flags. Ollama: terminal CLI plus HTTP API. LM Studio: graphical window plus optional HTTP server.
    • Model management. llama.cpp: you source and store the files. Ollama: curated registry plus import. LM Studio: in-app catalogue that stores models for you.
    • Control and tuning. llama.cpp: full (every flag is yours). Ollama: good (API and some options). LM Studio: good via sliders, less under the hood.
    • Best on CPU speed. llama.cpp is the most direct and measured the cleanest CPU numbers in the lab; the others are close.
    • Best for a headless server. Ollama, for the combination of easy install, scriptability and a standard API. llama.cpp if you want maximum control.
    • Best for a desktop / first time. LM Studio.

    Which one should you pick

    Match the tool to the situation rather than picking a “best” in the abstract:

    • You have a CPU box and you want the most performance with your hands on the controls: llama.cpp. Build it, load your model, set threads to your core count, and benchmark. It is the most capable and the least hand-holding.
    • You have a home server and you want a local model that is easy to install, easy to script, and friendly to the tools you already use: Ollama. It is the default for a reason, and it is what a chat front end like Open WebUI expects.
    • You are on a desktop and you want to try local AI now, with a window and no terminal: LM Studio. It is the friendliest, and it is the best place to learn which models and quantizations you actually prefer before committing to a server setup.

    And because the model files are the same GGUF across all three, you can start with the friendliest one and move to the more capable one without losing anything. A common path is to learn on LM Studio, then run the model you settled on with Ollama on the server, and reach for llama.cpp only when you need to tune or squeeze. The decision is about how you want to work, and the model you run is independent of that choice.

    Common mistakes

    Picking based on benchmarks you read rather than your own hardware. Published speed numbers depend on CPU, RAM, GPU and threads in ways that do not transfer. Run the benchmark on your box with the model you care about — llama.cpp’s benchmark tool does this directly, and Ollama reports timing in its API — and decide on your own numbers.

    Thinking you are locking yourself into a model ecosystem. You are not. The GGUF file is the constant; the runtime is swappable. If a tool stops fitting how you work, move the same model file to another one.

    Over-engineering the first setup. The fastest path to a useful local model is the least-friction tool you will actually keep using. Start with the one that matches your comfort (often Ollama on a server, LM Studio on a desktop), get one real answer back, and only then optimize toward llama.cpp if the speed or control genuinely requires it.

    How this fits the rest of your home server

    Whatever you pick here, the rest of the local AI stack is identical. The model is a GGUF file chosen by the size-and-quantization logic, the front end is a chat UI like Open WebUI, the security story is “keep the port private and reach it over your VPN” (the Tailscale guide), and the box you are running it on is the one you chose in the hardware guide. The runtime is the one swappable layer in the whole thing, and because it sits on top of a shared model format, the choice is a preference, not a commitment. Pick the front end that matches how you work, run a model you measured on your own hardware, and put a chat window in front of it — that is the whole stack, and the runtime is the only part you get to change your mind about freely.

    Tested on:

    OSDebian 12
    Hardware4-core / 16 GB, no GPU
    llama.cpp (4.65B Q4_K, 4 threads)48.4 t/s prompt / 13.6 t/s generation
    Ollama v0.34.0 (qwen3:0.6b Q4_K_M)~49-52 t/s generation

    Last tested: 15 September 2026

    What’s next?

    The natural next steps from this guide:

  • LM Studio: The Easiest Way to Run Local LLMs

    LM Studio: The Easiest Way to Run Local LLMs

    LM Studio is the friendliest on-ramp to local AI: a desktop app where you browse a catalogue of models, download one with a click, and start chatting in a window — no terminal, no config file, no “what is a context window” moment. It is the option to choose when you are working on a machine with a screen and you want the path of least resistance. Under the hood it runs the same kind of GGUF models that llama.cpp and Ollama use, and it can also switch on a local server so the rest of your tools can talk to it over the network. One honest note up front: this guide is written from the app’s documented behavior on macOS, Windows and Linux. The lab box for this series is a headless server without a desktop, so we have not screenshotted the GUI here; the steps and the ports and the server behavior are the documented ones, and the parts we can verify on a headless box — the server port, the OpenAI-compatible API — are what you will actually lean on if you run it on a server.

    Beginner · 9 min · Desktop app

    What LM Studio is and who it is for

    LM Studio is a graphical desktop application, available for macOS (including Apple Silicon), Windows and Linux. Its three jobs are:

    • Find and download models. It includes a search interface over a catalogue of GGUF models, so you can look at size, quantization and author, and download what fits your machine without hunting around a model host.
    • Chat with them. Once a model is downloaded, you pick it, set a few generation parameters with sliders, and talk to it in a chat window that behaves like a normal chat app.
    • Serve them. A “Local Server” toggle exposes the loaded model on http://localhost:1234 as an OpenAI-compatible API, so any client written for the OpenAI API can use it by changing the base URL. This is the part that turns a desktop app into a small model server.

    The model files it uses are the same GGUF files the other two runtimes load, so a model you download in LM Studio can be loaded in llama.cpp or Ollama too. There is no lock-in; LM Studio is a choice of interface, not a closed format.

    Installing it

    Download the installer for your platform from the LM Studio site and run it. On macOS and Windows it is a standard installer; on Linux it ships as an AppImage (or a platform package), which you make executable and run. After install, launch the app and you are in the interface. There is no separate “server daemon” to install on the desktop — the server is a feature you switch on inside the running app (see the server section), which is the thing to keep in mind if you are on a headless box: the desktop app expects a display, so on a server without a GUI the headless path is the one you want.

    Downloading your first model

    Inside the app, use the search/browse view to find a model. The two numbers to check are the parameter count and the quantization, exactly as in the size-and-quantization guide: pick something your RAM can hold. As a safe first download on a 16 GB machine, a 4B to 8B model at 4-bit is the same sweet spot recommended everywhere in this series. Select it and download. LM Studio stores models in a local models folder it manages for you, so you do not have to decide where the file lives or name it — a convenience, at the cost of less control over the exact file, which is the flip side of the friendliness.

    Chatting and tuning

    Select the downloaded model, and the chat window is ready. Above the input there are a handful of generation parameters — temperature, the context length, the maximum response length — exposed as sliders rather than flags. This is the main appeal for a first model: you can change how the model behaves and see the effect immediately, without editing a command. A good habit is to keep temperature modest (around 0.2 to 0.7) for factual or coding tasks and raise it only when you want more variety. The context slider is the one with a real cost: a longer context uses more memory, so if the model feels slow or the app warns about memory, that is the first knob to turn down.

    Turning it into a server on port 1234

    This is the step that makes LM Studio useful beyond the machine it is installed on. In the app, open the Local Server area and enable the server. It listens on http://localhost:1234 and exposes the loaded model through an OpenAI-compatible API. After that, any existing OpenAI client points at LM Studio by changing the base URL to that address and setting any API key value (the local server does not require a real key, but the clients expect the field to be filled). The endpoints line up with the ones you would use on the OpenAI API — a model list and a chat-completions endpoint — so the change is a configuration edit, not a rewrite. Two practical notes. First, the server reflects the model currently loaded in the app, so switching models in the UI changes what the server serves. Second, the server runs while the app runs; on a desktop that is fine, but it is also why the desktop app is the wrong shape for an always-on headless server, where you would want a service that starts at boot and has no window.

    Running it headless on a server

    If your target is a headless box (like most home servers), the desktop app is not the right vehicle, because it is built around a graphical window. There are two honest ways to run an LM-Studio-style setup headless. One is a community “headless server” package that runs the model server as a background service without the GUI and manages the model for you. The other — and the one we would actually recommend for a server — is to skip the GUI entirely and use a runtime that is native to the command line: Ollama or llama.cpp. The model files are the same GGUF files, so nothing is lost by moving from the desktop app to a headless runtime; you keep the models and change the front end. This is worth saying plainly: LM Studio shines on a desktop, and for a 24/7 server the command-line runtimes are the better fit, even though they all speak the same model format.

    Keeping it private

    The server defaults to localhost, which is the safe default: only the machine it runs on can reach it. If you want a chat app or another device on your network to use it, you can point the server at your LAN address, but the cleaner pattern is to keep the server on the box and reach it through your VPN — the Tailscale in Docker guide shows how to get a device on your home network from anywhere without opening a public port. And the same rule as the rest of this series: a local model server has no account system, so its security is entirely “who can reach the port”. Keep that on loopback or a private network, and if you do want a named, HTTPS front for the UI, the Caddy in Docker guide is the way to add it. The security guide has the firewall and network-isolation details.

    Common gotchas

    The server says the model is not available. The server serves whatever model is loaded in the app. If you downloaded a model but did not select it, or you switched models after starting the server, the API will not see the one you expect. Load the intended model, then start (or restart) the server.

    Port 1234 is already in use. If something else is bound to 1234, the server will not start or will collide. Change the port in the server settings, or free the port. This also bites when a desktop instance with the server on and a headless daemon are both running at once — two processes, one port. Pick one owner for the port.

    It is slower than Ollama or llama.cpp on the same model. The GUI and its management layer add overhead, and the desktop app is not optimized the way a bare command-line runtime is. For a desktop that is usually a non-issue; for a server where speed matters, the headless runtimes are the better tool.

    You expect the desktop app to run on a headless box. It is a GUI app. On a server with no display it will not do its job, which is exactly why the headless section above points you to a service or a command-line runtime instead.

    How this fits the rest of your home server

    LM Studio is the “I have a computer with a screen and I want to play with local AI today” option, and it is the best one for that job. It is also the place to start if you are deciding which models you actually like, because the one-click downloads and instant parameter changes make experimentation cheap. Once you know which model and which quantization you want to run for real — and you will, after a little playing — the natural move for a home server is a headless runtime that serves that same GGUF file around the clock: Ollama for the low-maintenance path, llama.cpp for the control-and-performance path. And whatever you run, the next step is the same: a chat front end the whole house can use, covered in the Open WebUI guide, and a bit of monitoring so the loaded model does not quietly take the RAM your other services need. The models are the constant across all of it; only the front end changes.

    What’s next?

    The natural next steps from this guide: