vLLM as an Always-On API

A containerised OpenAI-compatible endpoint on your LAN that survives reboots, uses the unified memory pool sensibly, and lets you swap the resident model without a rebuild.

Series 1, article 2 of 4 · written 31 July 2026

This is the article that turns the box from a thing you SSH into to a thing your other software talks to. The target is a single HTTP endpoint at http://spark.local:8000/v1 that speaks the OpenAI API, starts on boot, and serves several concurrent callers well — because concurrency is where this hardware is genuinely strong.

Why containers, specifically here

On a normal x86 workstation, running vLLM in a virtualenv is fine. On GB10 it is a trap. The stack you need — aarch64 wheels, CUDA 13, a vLLM built with sm_121a in its architecture list, a matching FlashInfer, and in practice a handful of patches — is fiddly to assemble and easy to break with a single pip install. Upstream vLLM support for sm_121 on aarch64 arrived late and has had gaps; the community has repeatedly done the source builds and published images.

So: use a container, and pick one whose author states which hardware they run it on.

# Verify the container toolkit is wired up before anything else
docker run --rm --gpus all nvidia/cuda:13.0.2-base-ubuntu24.04 nvidia-smi

DGX OS ships Docker and the NVIDIA Container Toolkit preinstalled. If nvidia-smi prints a GB10 inside the container, the plumbing is good.

A first launch

Start in the foreground so you can read the errors, and start with a model you know fits — a 25–35B class MoE at 4-bit is the right first target.

export MODELS=/mnt/models            # your model library root
export HF_HOME=$MODELS/hf

docker run --rm -it \
  --gpus all \
  --ipc=host \
  -p 8000:8000 \
  -v $MODELS:/models \
  -e HF_HOME=/models/hf \
  <your-gb10-vllm-image> \
  vllm serve /models/<model-dir> \
    --served-model-name daily \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 32768 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.75 \
    --max-num-seqs 16

The flags that matter, and why:

Do not pass tensor-parallel flags. You have one GPU; --tensor-parallel-size 1 is correct and is the default.

Checking it works

curl -s http://localhost:8000/v1/models | jq

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "daily",
    "messages": [{"role":"user","content":"In one sentence: what is unified memory?"}],
    "max_tokens": 100
  }' | jq -r '.choices[0].message.content'

From any other machine on the network, the same thing works against the Spark's hostname. Any OpenAI client library points at it by changing two variables:

from openai import OpenAI

client = OpenAI(base_url="http://spark.local:8000/v1", api_key="not-needed")

r = client.chat.completions.create(
    model="daily",
    messages=[{"role": "user", "content": "Summarise this changelog."}],
)
print(r.choices[0].message.content)

Making it permanent

Move from docker run to Compose, then let systemd own it.

# /opt/spark/vllm/compose.yml
services:
  vllm:
    image: <your-gb10-vllm-image>
    container_name: vllm
    restart: unless-stopped
    ipc: host
    ports: ["8000:8000"]
    volumes:
      - /mnt/models:/models
    environment:
      HF_HOME: /models/hf
    deploy:
      resources:
        reservations:
          devices: [{driver: nvidia, count: all, capabilities: [gpu]}]
    command: >
      vllm serve /models/${SPARK_MODEL}
      --served-model-name daily
      --host 0.0.0.0 --port 8000
      --max-model-len ${SPARK_CTX:-32768}
      --kv-cache-dtype fp8
      --gpu-memory-utilization ${SPARK_MEM:-0.75}
      --max-num-seqs ${SPARK_SEQS:-16}

# /opt/spark/vllm/.env
SPARK_MODEL=qwen-daily-nvfp4
SPARK_CTX=32768
SPARK_MEM=0.75
SPARK_SEQS=16

# /etc/systemd/system/vllm.service
[Unit]
Description=vLLM OpenAI-compatible server
Requires=docker.service
After=docker.service network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/spark/vllm
ExecStart=/usr/bin/docker compose up -d
ExecStop=/usr/bin/docker compose down
TimeoutStartSec=0

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now vllm
journalctl -u vllm -f

Swapping the resident model

vLLM serves one model per process and there is no graceful hot-swap. With one GPU and a one-model-at-a-time discipline, that is not a limitation so much as the design. A ten-line script makes it painless:

#!/usr/bin/env bash
# /usr/local/bin/spark-model — switch the resident vLLM model
set -euo pipefail
cd /opt/spark/vllm

if [[ $# -eq 0 ]]; then
  echo "resident: $(grep ^SPARK_MODEL .env | cut -d= -f2)"
  echo "available:"; ls -1 /mnt/models | sed 's/^/  /'
  exit 0
fi

[[ -d "/mnt/models/$1" ]] || { echo "no such model: $1" >&2; exit 1; }

sed -i "s|^SPARK_MODEL=.*|SPARK_MODEL=$1|" .env
[[ $# -ge 2 ]] && sed -i "s|^SPARK_CTX=.*|SPARK_CTX=$2|" .env

docker compose down
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null
docker compose up -d

echo -n "waiting for /v1/models "
until curl -sf localhost:8000/v1/models >/dev/null; do echo -n .; sleep 3; done
echo " ready"

spark-model                          # what is loaded, what is available
spark-model gpt-oss-120b-mxfp4       # switch to the heavyweight
spark-model qwen-daily-nvfp4 65536   # switch back, with a longer context

Because clients address the alias daily rather than a filename, nothing downstream needs to change when you switch. The cache drop between shutdown and start matters more than it looks: without it, the kernel is still holding the previous model's pages and the new load can hit an avoidable memory wall.

Confirming you actually got the concurrency

Single-request timing tells you almost nothing about this machine. Measure under load — vLLM ships a benchmark client:

docker exec -it vllm vllm bench serve \
  --backend openai-chat \
  --base-url http://localhost:8000 \
  --model daily \
  --dataset-name random \
  --random-input-len 1024 --random-output-len 256 \
  --max-concurrency 1   --num-prompts 20

# then repeat at 4, 8, 16, 32 and watch output throughput climb

What you should see is single-stream decode in the tens of tokens per second and aggregate output throughput climbing several-fold as concurrency rises, flattening once you saturate memory bandwidth or run out of KV cache blocks. If aggregate throughput does not climb, either --max-num-seqs is too low or the cache is full — check the vLLM logs for preemption warnings, which mean requests are being evicted and recomputed.

When it goes wrong

Sources

• NVIDIA DGX Spark Playbooks — official vLLM, TensorRT-LLM, SGLang and NIM walkthroughs.
• vLLM #31128 and #36821 — the sm_121 / aarch64 support history, worth reading before trusting an image.
• AEON vLLM Ultimate and vllm-gb10 — community images built for this silicon.
• AGmind — a documented GB10 stack with published TTFT, throughput and concurrency numbers.
• NVIDIA support — memory issues within capacity (the buffer-cache flush).