Chikewa

Start here

Search

Tag: llama.cpp

Runtime de código abierto para ejecutar LLMs localmente en CPU o GPU, con cuantización GGUF.

  • llama.cpp vs Ollama vs LM Studio: Which to Use

    llama.cpp vs Ollama vs LM Studio: Which to Use

    llama.cpp, Ollama and LM Studio all do the same fundamental job — load a GGUF model and generate text — and they all use the same model files, so choosing between them is not a question of capability. It is a question of how you want to work: how much control you want, how little setup you tolerate, and whether you are on a desktop or a headless server. This guide puts the three side by side, using real measurements from the lab box (a 4-core Intel i5-6500T, 16 GB RAM, Debian 12, no discrete GPU) wherever we could measure, and a clear recommendation for each situation. If you only read one thing: on a CPU box where speed matters, llama.cpp is the fastest; on a server where you want it to “just work” and be scriptable, Ollama is the best default; on a desktop where you want the friendliest experience, LM Studio wins.

    Intermediate · 11 min · Comparison

    The same engine, three front ends

    It is worth stating plainly that these are not three competing technologies. All three load GGUF model files and produce tokens. Ollama and LM Studio both build on the same inference foundation that llama.cpp provides, and they differ in packaging. That means:

    • The model files are interchangeable. A GGUF you use in one runs in the others. You are not choosing a model ecosystem, only a way to drive it.
    • The differences are in control, convenience and interface. llama.cpp exposes everything as flags. Ollama hides it behind a CLI and an API. LM Studio hides it behind a graphical window.
    • Speed differences are real but secondary. The engine is shared, so the gaps are in overhead and defaults, not in some deep architectural advantage. On a CPU box the direct llama.cpp path is the most efficient; the wrappers are close behind and the difference rarely matters compared to model choice.

    llama.cpp: control and pure-CPU performance

    llama.cpp is the bare engine you build and drive yourself. You build it once from source, then run a model with explicit flags for context, threads and host. What you get in return is the most direct path to the hardware and the best pure-CPU numbers. On the lab box we benchmarked a 4.65B model at Q4_K with four threads and measured about 48 tokens per second processing a 512-token prompt and about 14 tokens per second generating. Those are honest CPU figures for a modest box, and they are what you get when nothing sits between you and the engine.

    The cost is that there is no hand-holding: you find the model file yourself, you manage it, you set every parameter by hand, and the “server” is a process you start with a command. This is the right choice when you want to tune context length and threads, when you are on a CPU box where every token per second counts, or when you want to understand exactly what is running. It is the least friendly option, and the most capable one.

    Ollama: the low-maintenance default

    Ollama wraps the same engine in a single install and a tiny CLI, and adds a model registry that removes the “find the right file” step. ollama pull gets you a verified model; ollama run talks to it; and an HTTP API on port 11434 (plus an OpenAI-compatible endpoint) means every chat app and script can use it without custom code. On the lab box we pulled a 0.6B model at Q4_K_M (a 522 MB file, verified on download) and measured token generation at roughly 49 to 52 tokens per second and prompt processing at roughly 140 to 235 tokens per second. For a model that small those are “instant” numbers, and the point is less the raw speed than how little stood between us and a working answer.

    This is the best default for a home server: it installs cleanly, it is scriptable, it is what most third-party tools assume, and it stays out of your way. The trade-offs are that the registry is a curated list (a very specific self-downloaded file is imported rather than pulled by name), and it is a little less tunable than raw llama.cpp. For most people on a server, that trade is the right one.

    LM Studio: the friendliest desktop experience

    LM Studio is a graphical desktop app for macOS, Windows and Linux. You browse a model catalogue, download with a click, chat in a window, and adjust parameters with sliders. It is the fastest way to “have local AI on my computer today” with zero terminal, and the parameter sliders make experimenting with model behavior genuinely easy. It also serves the loaded model on port 1234 as an OpenAI-compatible API, so it can do the server job too. We have not screenshotted the GUI in this series because the lab box is headless, so treat the desktop specifics as the documented behavior; the parts that matter on a server — the port 1234 server and the OpenAI-compatible API — are the documented and verifiable ones.

    Where LM Studio gives ground is that it is a GUI app, so it is the wrong shape for an always-on headless server, and its management layer adds a little overhead compared to a bare command-line runtime. It is the best on-ramp and the best place to discover which models you like, and the natural next step from it is to hand the same GGUF file to a headless runtime for production. There is no lock-in, because the model format is shared.

    Side by side

    Summarizing what matters in a decision:

    • Setup. llama.cpp: build from source, manage files yourself. Ollama: one install script, one pull command. LM Studio: install an app, click to download. From most to least effort: llama.cpp, Ollama, LM Studio.
    • Interface. llama.cpp: terminal and flags. Ollama: terminal CLI plus HTTP API. LM Studio: graphical window plus optional HTTP server.
    • Model management. llama.cpp: you source and store the files. Ollama: curated registry plus import. LM Studio: in-app catalogue that stores models for you.
    • Control and tuning. llama.cpp: full (every flag is yours). Ollama: good (API and some options). LM Studio: good via sliders, less under the hood.
    • Best on CPU speed. llama.cpp is the most direct and measured the cleanest CPU numbers in the lab; the others are close.
    • Best for a headless server. Ollama, for the combination of easy install, scriptability and a standard API. llama.cpp if you want maximum control.
    • Best for a desktop / first time. LM Studio.

    Which one should you pick

    Match the tool to the situation rather than picking a “best” in the abstract:

    • You have a CPU box and you want the most performance with your hands on the controls: llama.cpp. Build it, load your model, set threads to your core count, and benchmark. It is the most capable and the least hand-holding.
    • You have a home server and you want a local model that is easy to install, easy to script, and friendly to the tools you already use: Ollama. It is the default for a reason, and it is what a chat front end like Open WebUI expects.
    • You are on a desktop and you want to try local AI now, with a window and no terminal: LM Studio. It is the friendliest, and it is the best place to learn which models and quantizations you actually prefer before committing to a server setup.

    And because the model files are the same GGUF across all three, you can start with the friendliest one and move to the more capable one without losing anything. A common path is to learn on LM Studio, then run the model you settled on with Ollama on the server, and reach for llama.cpp only when you need to tune or squeeze. The decision is about how you want to work, and the model you run is independent of that choice.

    Common mistakes

    Picking based on benchmarks you read rather than your own hardware. Published speed numbers depend on CPU, RAM, GPU and threads in ways that do not transfer. Run the benchmark on your box with the model you care about — llama.cpp’s benchmark tool does this directly, and Ollama reports timing in its API — and decide on your own numbers.

    Thinking you are locking yourself into a model ecosystem. You are not. The GGUF file is the constant; the runtime is swappable. If a tool stops fitting how you work, move the same model file to another one.

    Over-engineering the first setup. The fastest path to a useful local model is the least-friction tool you will actually keep using. Start with the one that matches your comfort (often Ollama on a server, LM Studio on a desktop), get one real answer back, and only then optimize toward llama.cpp if the speed or control genuinely requires it.

    How this fits the rest of your home server

    Whatever you pick here, the rest of the local AI stack is identical. The model is a GGUF file chosen by the size-and-quantization logic, the front end is a chat UI like Open WebUI, the security story is “keep the port private and reach it over your VPN” (the Tailscale guide), and the box you are running it on is the one you chose in the hardware guide. The runtime is the one swappable layer in the whole thing, and because it sits on top of a shared model format, the choice is a preference, not a commitment. Pick the front end that matches how you work, run a model you measured on your own hardware, and put a chat window in front of it — that is the whole stack, and the runtime is the only part you get to change your mind about freely.

    Tested on:

    OSDebian 12
    Hardware4-core / 16 GB, no GPU
    llama.cpp (4.65B Q4_K, 4 threads)48.4 t/s prompt / 13.6 t/s generation
    Ollama v0.34.0 (qwen3:0.6b Q4_K_M)~49-52 t/s generation

    Last tested: 15 September 2026

    What’s next?

    The natural next steps from this guide:

  • llama.cpp: Build, Run and Benchmark a Local LLM (CPU)

    llama.cpp: Build, Run and Benchmark a Local LLM (CPU)

    llama.cpp is the engine under the hood of most local AI setups, and it is also the most direct way to run a model when you have a CPU box and want to get the most out of it. It is a C++ program: you build it once from source (or grab a prebuilt binary), point it at a model file, and it generates text — from the command line, or as a small HTTP server that other apps can talk to. There is no account, no registry to sign up for, and nothing that phones home. This guide builds it, loads a real model, measures how fast it actually runs, and shows the exact flags that matter. Everything was done on a 4-core Intel i5-6500T with 16 GB of RAM and no discrete GPU, so the numbers here are what a modest CPU box really delivers.

    Intermediate · 12 min · C++ / CLI

    What llama.cpp gives you

    When you build llama.cpp you get a family of small tools in a single build/bin folder. The ones you will actually use are:

    • llama-server — runs a model as an HTTP server with an OpenAI-compatible API. This is what you point a chat app or a script at. It is the one we used in the lab.
    • llama-cli — an interactive chat in your terminal. Good for a quick “does this model work” check.
    • llama-bench — measures prompt-processing and token-generation speed for a model. This is how we got the numbers below, and it is the tool to reach for whenever you want to compare models or quantizations objectively.
    • llama-quantize — converts a full-precision model to a smaller quantized version (Q4, Q5, Q8, …). Useful when you have a full model and want to shrink it to fit your memory.

    The reason people use it directly rather than a wrapper is control: you choose the context length, the number of threads, the GPU split, and you see exactly what is happening. The cost is that there is no one-click “pull this model” — you manage the model files yourself.

    Building it

    The build needs a C++ compiler and CMake. On a Debian/Ubuntu box:

    sudo apt update
sudo apt install -y build-essential cmake git

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)

    On the lab box this completed cleanly with the default CPU backend. The key point for a CPU-only machine: leave GPU support off (which is the default when you have no CUDA toolkit installed). If you do have an NVIDIA GPU, you install the CUDA toolkit and add -DGGML_CUDA=ON to the cmake -B build line, and the same build will offload the model weights to the GPU. We built a CPU Release build here, which is the configuration that produces the honest CPU numbers in this guide.

    After the build, everything you need is in build/bin. Confirm it with ./build/bin/llama-cli --version. In the lab we were on a recent build (commit dd1ea5243), which is what produced the binary that ran the tests below.

    Loading and running a model

    A model for llama.cpp is a .gguf file — one file, self-contained, with the weights and the chat template inside it. You get these from a model host such as Hugging Face (the GGUF versions), and you put them anywhere on disk. You do not need a specific folder structure; the tools just take a path.

    For a quick interactive test, llama-cli is the fastest way to see if a model works at all:

    ./build/bin/llama-cli -m /path/to/model-Q4_K_M.gguf -c 4096 -t 4

    The flags that matter:

    • -m — the model file. Everything else hangs off this.
    • -c — the context window in tokens. This is how much of the conversation the model can “see” at once. It costs memory, so do not set it larger than you need. 4096 is a fine default; we used 32768 for the server run below.
    • -t — the number of CPU threads. Match this to your physical cores. On the 4-core lab box, -t 4 is the honest choice; setting it to 8 would just add contention.

    Running it as a server

    For real use, run llama-server so your apps can talk to it over HTTP. In the lab this is exactly the command we used:

    ./build/bin/llama-server \
  -m /home/chikewa/models/gemma-4-E2B-it-UD-Q4_K_XL.gguf \
  --host 0.0.0.0 --port 8080 \
  -c 32768 \
  --api-key-file /home/chikewa/api-keys.txt

    This serves an OpenAI-compatible API on port 8080. Once it is up, the model list lives at /v1/models and chat completions at /v1/chat/completions, so any client written for the OpenAI API works unchanged. We confirmed the server answered {"status":"ok"} on its health endpoint and returned the loaded model on /v1/models before sending any real requests. Two notes from the lab: bind --host 0.0.0.0 only if you genuinely intend the rest of the network to reach it; on a server that should stay private, --host 127.0.0.1 is the safer default and you reach it through your VPN. And the --api-key-file flag means the server will require a key on requests, which is worth turning on the moment anything other than your own machine might see the port.

    Measuring how fast it actually is

    This is the part that turns a guess into a decision. llama-bench runs a prompt-processing test (how fast it reads your input) and a token-generation test (how fast it produces the answer), and prints tokens per second for each. On the lab box, with the 4.65B-parameter gemma model at Q4_K and -t 4:

    ./build/bin/llama-bench \
  -m /home/chikewa/models/gemma-4-E2B-it-UD-Q4_K_XL.gguf \
  -p 512 -n 128 -t 4

    The result we measured:

    • 512-token prompt processing: about 48.4 tokens per second. This is the “it reads your long prompt” speed. At ~48 t/s, a 1,000-token prompt is digested in roughly 20 seconds.
    • 128-token generation: about 13.6 tokens per second. This is the “it writes the answer” speed — the number you feel. 13.6 t/s means a 200-word answer streams back in around 30 seconds, which is genuinely usable for everyday tasks even though it is not instant.

    These two numbers are what you want to capture for your own model and your own hardware, because they are the ground truth for whether a bigger model is worth the memory. Run llama-bench on the model you are about to commit to, and if generation drops below a pace you find pleasant, the fix is usually a smaller model or a higher quantization trade-off — or a GPU, which is the single biggest jump available.

    Common gotchas

    It is slower than the benchmarks claim. The most common cause is the context length. -c is not free: a 32k context holds a lot of state in memory and can slow generation noticeably compared to a 4k context, especially on CPU. If a model is “supposed” to be fast but is not, halve the context and re-benchmark before blaming the model. The second cause is thread count set above your physical cores, which adds contention rather than speed.

    The model does not fit, or the box swaps. If you load a model bigger than your available RAM, the operating system will start swapping to disk and the speed will collapse from tens of tokens per second to a trickle. Watch memory while it loads; if free RAM goes negative and swap climbs, the model is too big for the box at that quantization. The fix is a smaller model or a lower quant, per the size-and-quantization guide.

    The server answers but my client gets an empty response. With a few newer “thinking” models, the model spends its first tokens on internal reasoning that is returned in a separate field rather than in the main answer. If your client only reads the primary content field, it can look like the model returned nothing even though it worked. This is a model-behavior detail, not a llama.cpp bug — check the full response for a separate reasoning field before assuming the server is broken. We hit this with a thinking model in the lab, which is why it is worth knowing.

    Two versions of the binaries, or a stale build. If you rebuilt and the old behavior persists, you are almost certainly running the old binary from a different path. which llama-server and a fresh --version tell you exactly which build you are executing.

    How this fits the rest of your home server

    llama.cpp is the “I want the model itself, with my hands on it” option in the local AI stack. It is the right first choice on a CPU box because it is the most efficient of the three runtimes on that hardware, and it is the one to reach for when a wrapper is hiding something you want to change. The model files it loads are the same GGUF files that Ollama and LM Studio use, so nothing here is a dead end — if you later want the one-click model downloads or a nicer desktop, you move the runtime and keep the models. For the “just give me a model” path, see the Ollama guide; for the desktop route, the LM Studio guide; and for putting a proper chat window in front of whatever you run, the Open WebUI guide. And because a loaded model will happily hold several gigabytes of RAM, it belongs on a box you are watching — the Prometheus setup is the natural companion, so a chatty model does not quietly crowd out the rest of your services.

    Tested on:

    OSDebian 12
    HardwareIntel i5-6500T, 4 cores / 16 GB, no GPU
    llama.cppbuild dd1ea5243 (b10343-12), CPU Release build
    Model4.65B-param model at Q4_K (2.95 GB)
    Measured speed48.4 t/s prompt (512) / 13.6 t/s generation (128), 4 threads

    Last tested: 15 September 2026

    What’s next?

    The natural next steps from this guide:

  • GGUF and Quantization Explained: Pick the Right Model Size

    GGUF and Quantization Explained: Pick the Right Model Size

    Every local model you run is described by a name like some-model-8B-Q4_K_M.gguf, and almost everyone treats those bits after the name as magic strings. This guide takes them apart. What is a GGUF file, what do the two numbers in a model name actually mean, what a quantization level really costs in memory and quality, and how to pick one for the box you have. It is the reference the rest of the series points to, because once you understand size and quantization, “which model should I run” stops being a guess and becomes arithmetic.

    Beginner · 10 min · Concepts

    What a GGUF file is

    GGUF is a single-file format for storing a model. Inside one .gguf file you get the model’s weights, the architecture description, the chat template (the instructions that tell the model how to format a conversation), and the vocabulary. That single-file property is the whole point: there is no folder of parts to assemble, no separate tokenizer to match, nothing to forget. You copy one file from machine to machine, or download one file, and it runs. llama.cpp, Ollama and LM Studio all read it, which is why the model ecosystem is built on it. A model you download as GGUF for one of those tools is the same file that the others load — the format is the common currency, and the tools are just different front ends for it.

    The two numbers in a model name

    Take a name like model-8B-Q4_K_M. The 8B is the parameter count: roughly eight billion parameters. A parameter is a number the model learned, and the count is a coarse measure of how much it can hold in its “head”. More parameters generally means more capability, but also more memory and slower generation, all else equal. The Q4_K_M is the quantization, and it is the second, equally important half of the name. You will see the parameter count written as 0.6B, 1.5B, 4B, 8B, 14B, 32B, 70B and so on. As a rule of thumb for the “B” number: the smaller ones are fast and lightweight, the ones in the 4B to 14B range are the current sweet spot for home hardware, and the 32B and 70B models are the high end that want a real GPU or a lot of RAM.

    What quantization means

    A model’s parameters are originally stored at high precision, typically 16-bit or 32-bit numbers. Storing and computing on those is accurate but heavy. Quantization is the process of storing each parameter with fewer bits, trading a little precision for a lot less memory and (often) faster computation. The “Q” in the name stands for the number of bits per parameter: Q8 is 8 bits, Q4 is 4 bits, and so on. Going from 16 bits down to 4 bits shrinks the file to a quarter of its size. The cost is that each parameter is now an approximation of the original, and the model’s output quality degrades a little as you compress harder. Quantization is what makes running a 70B model at all possible on consumer hardware — at full precision it would need hundreds of gigabytes, but quantized it can fit in a fraction of that.

    Reading the quantization codes

    The codes you see in practice, from highest to lowest quality, are roughly:

    • F16 / F32 — full precision. The original, uncompressed values. Used for training and as a starting point; far too big to run locally.
    • Q8_0 — 8-bit. Very close to full precision in quality, and about half the size. The “safe” high-quality option when your memory allows it.
    • Q6_K — 6-bit, a “K-quant”. A good quality/size compromise.
    • Q5_K_M — 5-bit. Often the best balance for models you want to run well without huge memory.
    • Q4_K_M — 4-bit. The most common default and the one to understand. The “_K” means it is a mixed-precision K-quant: some parts of the model are kept at slightly higher precision than others, which preserves more quality than a flat 4-bit would. The “_M” is a size variant of that scheme. Q4_K_M is the quantization most model hosts offer first, and for good reason: it is small enough to run on modest hardware and good enough that the quality loss is often barely noticeable.

    There are also lower options (Q3, Q2 and below) that squeeze the model much smaller at a real quality cost, and they are worth using when the alternative is not being able to run the model at all. The practical guidance is: reach for the highest quantization your memory can hold. If a model’s Q5 does not fit but its Q4 does, run the Q4. Do not run a Q4 when your box could hold the Q5 — the extra quality is free if you have the memory.

    The memory math, with real numbers

    Here is the arithmetic that turns a model name into a memory requirement. The weight size in memory is approximately the parameter count times the bits per parameter, divided by 8 to get bytes. Worked out for a few real cases on the kind of hardware in this series:

    • A 4.65B model at Q4 is about 2.95 GB of weights. That is the exact model we ran in the lab: it loaded into a 16 GB box with room to spare for a 32k context, and generated at about 14 tokens per second on a 4-core CPU. The file was 2.95 GB on disk, and the loaded model fit comfortably with several gigabytes of RAM left over for the rest of the system.
    • A 0.6B model at Q4_K_M is about 0.5 GB. This is the model we used for the Ollama tests; it loaded and generated almost instantly, which is why starting with a small model is the right first move.

    Two things to add to the weight size. First, the context window costs memory on top of the weights, and it grows with the context length, so a 32k context needs more RAM than a 4k one for the same model. Second, if you have a GPU, you want the weights to fit in VRAM, not system RAM — an 8 GB card can hold roughly an 8B model at Q4 plus a modest context, which is why “8B at 4-bit” is the number people aim for on an 8 GB GPU. The whole exercise is the same everywhere: count the parameters, apply the bits, add the context, and check it fits the memory you actually have. If it does not, drop the quantization or the model size, not your expectations about the rest of the box.

    How to pick a quantization for your box

    A method that avoids most dead ends:

    1. Find your memory budget. Free system RAM for a CPU box, or VRAM for a GPU. This is the ceiling.
    2. Reserve room for context and the OS. Do not spend the entire budget on weights. Leave a couple of gigabytes for the context and the rest of the system, or you will be swapping.
    3. Pick the model size first, then the highest quant that fits. Decide how smart you need it to be (the “B” number), then take the best quantization your remaining memory allows. Q8 if you can, else Q6, Q5, Q4 in that order.
    4. Prefer a smaller model at a higher quant over a bigger one at a lower one, up to a point. A 7B at Q5 often beats a 13B at Q3, because the lower quant degrades quality faster than the extra parameters add it. There is a crossover where the bigger model wins, but for home hardware the smaller-and-cleaner choice is usually right.
    5. Measure, do not assume. Run the model, note the tokens per second, and judge the output on a few real tasks. The numbers tell you whether the quantization you picked is the right trade for your use.

    Common mistakes

    Picking by parameter count only. “70B is the best, I want 70B” is how people end up with a model that cannot load on their box. The parameter count and the quantization are one decision, not two. Always do the memory math before you commit to the size.

    Running a low quant when a higher one fits. If your box can hold the Q5, running the Q4 is leaving quality on the table for free. Check the memory, then take the best quant that fits.

    Setting a huge context on a small box. The context is real memory, and it is the quiet thing that pushes a “should fit” model into swapping. If a model loads but is mysteriously slow, the context length is the first thing to cut.

    Mixing a model and quantization from different sources without checking they match. The GGUF file already has its quantization baked in, so a mismatch is not really possible within a single file — but it is worth knowing that the quantization is a property of the file you download, not a setting you apply later. You cannot “quantize down” a file you already have without running a quantization tool on the full-precision weights.

    How this fits the rest of your home server

    Size and quantization are the vocabulary that connects every other guide in this series. When the llama.cpp guide tells you to check whether a model fits, this is the math it means. When the Ollama guide says to start with a small model, this is why 0.5 GB loads instantly and 2.95 GB is the sweet spot on a 16 GB box. When you are deciding whether a bigger model is worth it, the tokens-per-second number you measure is the answer to the question this guide sets up. The model file is the constant: pick the size and quantization that fit your memory, download it once as a GGUF, and any of the runtimes can load it. That is the entire decision, reduced to arithmetic and one measurement.

    Tested on:

    Hardware4-core / 16 GB
    4.65B model at Q4_K2.95 GB on disk, runs in 16 GB RAM with a 32k context
    0.6B model at Q4_K_M522 MB on disk, loads instantly

    Last tested: 15 September 2026

    What’s next?

    The natural next steps from this guide: