The lead article: memory arithmetic, measured throughput, and choosing a serving engine before you download sixty gigabytes of weights.
The marketing line for this class of machine is "run 200-billion-parameter models locally." That is true, and it is also the least useful sentence you will read about it. The question that matters is not what the box can hold but what it can hold while still being pleasant to use — and those are different numbers.
Three things consume the pool: the weights, the KV cache, and everything else. The weights are easy to estimate:
footprint ≈ parameters × bits_per_parameter / 8At 4-bit that is roughly half a gigabyte per billion parameters; at 8-bit, a gigabyte per billion. The KV cache is the part people forget, and on long contexts it can rival the weights. It scales with context length, batch size and the model's attention geometry — for a big model at a long context it is routinely tens of gigabytes. In one documented GB10 deployment serving a 26B mixture-of-experts model at 65K context, the split was 48.5 GiB of weights against 41.7 GiB of KV cache: the cache was nearly as large as the model.
Budget conservatively. Of the 128 GB, about 120 GB is realistically addressable, and a working deployment should leave several gigabytes of headroom for activations and fragmentation.
| Model class | Weights at 4-bit | Verdict on a GX10 |
|---|---|---|
| ~8–14B dense | 4–7 GB | Trivial. Use the leftover memory for a very long context, or run several at once. |
| ~27–35B dense or small MoE | 14–24 GB | The sweet spot. Fast enough to feel interactive, room for a big KV cache. |
| ~70B dense | ~35–40 GB | Comfortable on memory, slow on generation — roughly 35–45 tok/s single stream. |
| ~120B sparse MoE | ~60 GB | Fits well and performs better than the parameter count suggests, because only a fraction of the weights activate per token. |
| ~200B+ MoE | ~100 GB+ | Loads, runs, leaves almost nothing for KV cache. Short contexts only, and you will feel every token. |
| 400B–1T+ MoE | 200 GB+ | Does not fit on one box. This is where the honest answer is "not on this machine." |
Published measurements on GB10 hardware, gathered from the llama.cpp benchmark thread, NVIDIA's developer forums and community deployments. Treat them as the right order of magnitude rather than a promise — engine versions move fast, and the gap between a naive and a tuned configuration is large.
| Configuration | Prefill (prompt) | Decode (generation) |
|---|---|---|
| gpt-oss-20b, MXFP4, Ollama | ~2,050 tok/s | ~50 tok/s |
| gpt-oss-120b, MXFP4, llama.cpp | ~1,700–1,800 tok/s @2K ctx | ~35–46 tok/s |
| gpt-oss-120b, MXFP4, vLLM | — | ~59 tok/s |
| ~30B coder MoE, llama.cpp | >1,600 tok/s | 26–44 tok/s |
| ~26B MoE, NVFP4 + speculative decode, vLLM | — | ~50 tok/s single; ~1,430 tok/s aggregate |
| ~26B MoE, fp8 KV, 65K ctx, vLLM | 183 ms TTFT | 23–24 tok/s single; 45 parallel requests sustained |
Two observations do most of the explanatory work here.
Prefill is fast; decode is not. Reading a prompt is compute-bound and the Blackwell GPU is good at it — a thousand-plus tokens per second is normal. Writing the answer is bandwidth-bound and lands in the tens of tokens per second. So the box is well suited to workloads that read a lot and write a little — summarisation, classification, retrieval-augmented answering, code review — and poorly suited to generating long documents while you wait.
Aggregate throughput is where it wins. The 50 tok/s single-stream figure and the 1,430 tok/s aggregate figure are the same model on the same machine. Single-stream decode leaves the GPU mostly idle waiting on memory; concurrent requests fill those gaps almost for free. If your use case is several agents, a household of users, or a batch job over a thousand documents, this hardware is far better than the single-user benchmarks suggest.
Long context is where enthusiasm meets arithmetic. A user on NVIDIA's forums measured prompt processing on a 30B-class model across context depths and found throughput falling from roughly 1,880 tok/s at 8K context to about 425 tok/s at 128K — and extrapolated to under 1 tok/s approaching 500K. Attention cost grows with sequence length, and the KV cache grows linearly on top of it, eating the memory you were relying on.
The practical rule: a 32K–64K working context is comfortable and fast. 128K is usable if you are patient and have budgeted the cache. The million-token context windows in model marketing are, on this machine, a specification rather than a workflow. If you need to reason over more text than that, retrieval is the answer, which is series 4.
You will meet four names and they are not interchangeable.
| Format | What it is | When to use it on GB10 |
|---|---|---|
| NVFP4 | NVIDIA's 4-bit floating point with hardware acceleration on Blackwell | The native choice. Fastest path on this silicon; used by most GB10-targeted vLLM and TensorRT-LLM deployments. |
| MXFP4 | Open microscaling 4-bit format | What several major models ship in natively. Well supported, slightly behind NVFP4 on tuned paths. |
| GGUF (Q4_K_M, Q8_0, …) | llama.cpp's quantized container format | Widest model selection by far, easiest to obtain, best for the interactive/experimental path. |
| FP8 / BF16 | 8-bit and 16-bit | BF16 for anything under ~30B where you want maximum fidelity. FP8 is most valuable applied to the KV cache rather than the weights. |
NVIDIA's own position is that NVFP4 plus speculative decoding is where the recent performance gains came from — the company claims to have more than doubled DGX Spark throughput since launch through software alone, citing up to a 2.6× improvement over FP8 execution on a large MoE model. That claim is directionally consistent with what community deployments report.
All five major engines run on this hardware. They are genuinely different tools, and the choice matters more than the model choice for how the box feels.
| Engine | Reach for it when | Cost |
|---|---|---|
| vLLM | You want a persistent OpenAI-compatible API serving several clients. Best-in-class continuous batching — this is how you get the aggregate throughput. | Heavier to set up on sm_121a; needs a GB10-aware build. Model switching means a restart. |
| llama.cpp | One person, one conversation, many different models, minimum ceremony. Widest model availability via GGUF. | Weaker under concurrency than vLLM. |
| Ollama | You want something working in ten minutes and do not want to think about flags. | Least control; leaves performance on the table. |
| SGLang | Structured generation, complex multi-step agent prompting, heavy prefix reuse. | Smaller community on this hardware than vLLM. |
| TensorRT-LLM | You want the maximum single-stream tokens per second and will spend a day compiling engines to get it. | Build complexity, per-model engine compilation, rigid once built. |
The rule of thumb the community has converged on: concurrency → vLLM or SGLang; one local user → llama.cpp; squeeze the last token per second out of one stream → TensorRT-LLM. This series builds the first two, in articles 2 and 3, because between them they cover the API and the UI without requiring a compilation project.
Any specific list of model names written today will be partly wrong by Christmas. What is durable is the shape of a sensible library, so build it by role rather than by brand:
If you want the box to be a service that other things call — your editor, your scripts, an agent framework — start with vLLM as an always-on API. If you mostly want to talk to a model in a browser and swap models often, start with llama.cpp and Open WebUI. Most people end up running both: vLLM for the model that is always on, llama.cpp for experiments.