llama.cpp and Open WebUI: the Interactive Path

Building for compute capability 12.1, the mmap question that has the opposite answer here than on a discrete card, automatic model swapping, and a browser UI on the LAN.

Series 1, article 3 of 4 · written 31 July 2026

vLLM is the right tool for a model that is always on. It is the wrong tool for the evening you want to try six models, three of which you will delete. That is llama.cpp's job: a single binary, a directory of GGUF files, and no infrastructure. Paired with Open WebUI it also gives you the browser interface — chat history, document upload, multiple models in a dropdown — that makes the box useful to people who are not going to type curl.

Building llama.cpp for GB10

Prebuilt binaries generally will not have the right CUDA architecture compiled in. The build is quick and unfussy, and it is one of the few places on this machine where compiling from source is genuinely the easy path.

sudo apt update && sudo apt install -y build-essential cmake git libcurl4-openssl-dev

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=121 \
  -DLLAMA_CURL=ON \
  -DCMAKE_BUILD_TYPE=Release

cmake --build build --config Release -j"$(nproc)"

CMAKE_CUDA_ARCHITECTURES=121 is the whole trick — compute capability 12.1 is the GB10 Blackwell GPU. Leave it out and you either get a build that will not run or one that falls back to a generic path and runs slowly. Confirm afterwards:

./build/bin/llama-cli --version
./build/bin/llama-bench -m /mnt/models/gguf/<model>.gguf -p 512 -n 128

llama-bench printing prompt-processing figures in the high hundreds or thousands of tokens per second means the GPU is doing the work. Double digits means it is not, and you are running on the Grace cores.

The mmap question

By default llama.cpp memory-maps model files, so the kernel pages weights in as they are touched. On a discrete GPU this is a clean optimisation. On a unified-memory machine the picture is muddier, and you will find advice pointing in both directions — this is not people being careless, it is two different problems.

For llama.cpp, contributors on the GB10 benchmark thread found that disabling mmap improved consistency: the mapped pages and the model's resident copy end up double-counted against the same pool, and the kernel's caching heuristics were tuned for a world where GPU memory is somewhere else.

For ComfyUI (covered in the video series), the GB10-specific guidance is the opposite — --disable-mmap there forces a full read-and-copy of a very large checkpoint at load time and makes things worse.

The reconciliation: mmap hurts when a process maps a file and holds a separate copy; it helps when it avoids a copy entirely. Measure on your own workload. For llama.cpp on this box, start here:

./build/bin/llama-bench -m model.gguf -p 2048 -n 256 --mmap 1
./build/bin/llama-bench -m model.gguf -p 2048 -n 256 --mmap 0

and, either way, drop the page cache between runs so you are not measuring the previous test's leftovers.

llama-server: the API side

llama.cpp ships its own OpenAI-compatible server. It is lighter than vLLM and perfectly adequate for one or two users.

./build/bin/llama-server \
  -m /mnt/models/gguf/daily.gguf \
  --host 0.0.0.0 --port 8080 \
  --ctx-size 32768 \
  --n-gpu-layers 999 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --parallel 4 \
  --no-mmap

It serves both a built-in chat page at http://spark.local:8080 and an OpenAI-compatible API at /v1. For quick experiments the built-in page is often all you need.

Automatic model swapping

The friction in the experimental workflow is that every model change means stopping and restarting the server with a new path. A model-swapping proxy removes it: it sits on one port, watches which model each request asks for, and starts or stops llama-server processes on demand. llama-swap is the established one.

# /opt/spark/llama-swap/config.yaml
healthCheckTimeout: 300

models:
  daily:
    cmd: >
      /opt/llama.cpp/build/bin/llama-server
      -m /mnt/models/gguf/daily.gguf
      --port ${PORT} --ctx-size 32768
      --n-gpu-layers 999 --no-mmap
    ttl: 900

  heavy:
    cmd: >
      /opt/llama.cpp/build/bin/llama-server
      -m /mnt/models/gguf/heavy-120b.gguf
      --port ${PORT} --ctx-size 16384
      --n-gpu-layers 999 --no-mmap
    ttl: 300

  draft:
    cmd: >
      /opt/llama.cpp/build/bin/llama-server
      -m /mnt/models/gguf/small-4b.gguf
      --port ${PORT} --ctx-size 8192
      --n-gpu-layers 999
    ttl: 1800

Request "model": "heavy" and the proxy unloads whatever is resident, loads the 120B, and answers. The ttl unloads an idle model so it is not squatting on 60 GB while you are asleep. This enforces the one-model-at-a-time discipline mechanically instead of by memory and willpower.

Open WebUI

Open WebUI is the browser front end: conversations, model picker, prompt library, document upload, multi-user accounts. It talks to any OpenAI-compatible backend, so it sits equally happily on top of llama-swap, llama-server or the vLLM endpoint from article 2.

# /opt/spark/openwebui/compose.yml
services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports: ["3000:8080"]
    volumes:
      - ./data:/app/backend/data
    environment:
      # point at whichever backend is authoritative
      OPENAI_API_BASE_URL: http://host.docker.internal:8080/v1
      OPENAI_API_KEY: not-needed
      # turn off the phone-home and the default local embedding downloads
      ENABLE_OLLAMA_API: "false"
      WEBUI_AUTH: "true"
    extra_hosts:
      - "host.docker.internal:host-gateway"

cd /opt/spark/openwebui && docker compose up -d
# then browse to http://spark.local:3000 and create the first account

The first account created becomes the administrator. Leave WEBUI_AUTH on even on a home network — it costs one login and it stops a guest device from wandering into your chat history.

Open WebUI also has built-in retrieval features. They work, and they are the fastest way to ask questions about a handful of documents. They are also deliberately simple, and if documents are the point rather than a convenience, series 4 builds something considerably better.

Ollama, and when it is the right answer

Ollama wraps llama.cpp with a model registry and automatic memory management. It is the ten-minute path, it is packaged for the Spark, and its API is what a great deal of third-party software expects.

curl -fsSL https://ollama.com/install.sh | sh

# keep the model store off the internal drive
sudo systemctl edit ollama
#   [Service]
#   Environment="OLLAMA_MODELS=/mnt/models/ollama"
#   Environment="OLLAMA_HOST=0.0.0.0"

ollama pull <model>
ollama run  <model>

The trade is control. Ollama picks its own quantization defaults, its own context length and its own offload policy, and on this hardware those defaults leave real performance on the table — published GB10 comparisons show Ollama trailing a tuned llama.cpp or vLLM configuration by a wide margin. Use it to get started or to satisfy software that only speaks Ollama; move to llama.cpp or vLLM when the speed starts to matter.

Which to run

The configuration that most people converge on, and the one this series recommends:

• vLLM on :8000 — the one model that is always on, serving scripts, agents and editors, tuned for concurrency.
• llama-swap on :8080 — the experimental bench, models loading and unloading on demand.
• Open WebUI on :3000 — pointed at both, so the model dropdown offers everything.

All three are idle-cheap. The rule that keeps the memory arithmetic honest is that only one large model is resident at a time: llama-swap's TTLs handle that automatically, and switching the vLLM model is the spark-model script from the previous article.

Sources

• Performance of llama.cpp on NVIDIA DGX Spark — build flags, mmap findings, measured throughput.
• NVIDIA DGX Spark Playbooks — official llama.cpp, Ollama and Open WebUI walkthroughs.
• llama-swap — on-demand model loading proxy.
• Open WebUI.
• llama.cpp-dgx — a GB10-targeted fork, for when a baseline is not enough.