Quantization formats that matter on Blackwell, why speculative decoding triples single-stream speed and then evaporates under load, and a measurement habit so you are tuning against numbers instead of impressions.
NVIDIA's claim that DGX Spark performance more than doubled between launch and CES 2026 — with no hardware change — is true and is also the most important thing to understand about tuning this box. Nothing about the silicon improved. What improved was the software's use of a 4-bit numeric format the hardware accelerates natively, and a decoding strategy that stops wasting memory bandwidth. Both are things you have to opt into.
Every optimisation below is worth between nothing and 3.5×, depending on the model, the engine and the workload. The only way to tell which is to measure, and the only measurement that means anything is one taken twice — before and after a single change.
# A three-number baseline, taken the same way every time.
# 1. Single-stream decode: what one person feels.
# 2. Prefill: how fast it reads a long prompt.
# 3. Aggregate under concurrency: what the box is actually worth.
# llama.cpp
./build/bin/llama-bench -m model.gguf -p 2048 -n 256 -r 3
# vLLM, sweeping concurrency
for c in 1 4 8 16 32; do
vllm bench serve --backend openai-chat --base-url http://localhost:8000 \
--model daily --dataset-name random \
--random-input-len 1024 --random-output-len 256 \
--max-concurrency $c --num-prompts $((c*5)) \
| grep -E 'Output token throughput|Mean TTFT|Mean TPOT'
doneDrop the page cache between runs (sync; echo 3 > /proc/sys/vm/drop_caches) or you will measure the previous configuration's warm pages and conclude that whatever you did last was brilliant.
Blackwell has hardware acceleration for 4-bit floating point. Using a format the silicon does not accelerate means doing the dequantisation work in software, which is exactly the tax you were trying to avoid.
| Format | Character | Use it for |
|---|---|---|
| NVFP4 | NVIDIA 4-bit float, accelerated on Blackwell via CUTLASS paths | The default for anything served through vLLM or TensorRT-LLM on this box. Best speed per gigabyte. |
| MXFP4 | Open microscaling 4-bit; the native format of several major releases | Use it when the publisher shipped it — converting to NVFP4 for a marginal gain is rarely worth the quality risk. |
| GGUF Q4_K_M | llama.cpp's workhorse 4-bit | Experimentation, breadth of model choice, the interactive path. |
| GGUF Q8_0 / BF16 | 8- and 16-bit | Models small enough that you have memory to spare and want maximum fidelity — under ~30B. |
The quality cost of 4-bit is real but small, and it is not uniform: it shows up first in long-chain arithmetic, code that must compile, and strict format adherence. If a model is failing at structured output, try the same model one quantization step up before concluding the model is wrong for the job.
This is the most under-used lever on a memory-constrained machine, because the cache can be as large as the model. In a documented GB10 deployment at 65K context, a 26B MoE occupied 48.5 GiB of weights and 41.7 GiB of KV cache. Halving the second number is worth more than most weight-level optimisation.
# vLLM
--kv-cache-dtype fp8
# llama.cpp
--cache-type-k q8_0 --cache-type-v q8_0FP8 or Q8 cache is close to free in quality terms. More aggressive cache quantization exists — community work on GB10 reports KV compression around 3.9× — and starts to be noticeable on long documents. Try it when you need the context length; do not adopt it by default.
Decoding one token at a time means streaming the model's active weights through memory once per token. That is the bandwidth wall. Speculative decoding attacks it directly: something cheap proposes several tokens, and the real model verifies them all in a single forward pass. When the guesses are right, you got several tokens for the bandwidth cost of one.
Two families are in play on this hardware:
The measured effect is substantial. A 31B-class model on GB10 was reported going from 11 tok/s to 39 tok/s single-stream — about 3.5× — with NVFP4 plus DFlash. NVIDIA separately reports up to 2.6× over FP8 execution on a large MoE using NVFP4 and speculative decoding together.
With a draft model, in vLLM, the shape is:
vllm serve /models/daily \
--speculative-config '{
"model": "/models/small-draft",
"num_speculative_tokens": 5
}' \
--max-num-seqs 1 \
...The draft model must share a tokenizer with the target and should be roughly an order of magnitude smaller. Acceptance rate is what determines the payoff: a draft that is frequently wrong makes things slower, not faster. vLLM logs the acceptance rate — watch it, and if it is poor, either pick a better-matched draft or turn the feature off.
For the block-diffusion path, the implementation is bundled into specific GB10 images rather than being an upstream flag; follow the deployment notes of whichever image you are running, since the configuration keys differ between them.
It is tempting to set the maximum context to whatever the model advertises. On this machine that decision costs memory you did not intend to spend and speed you will not get back — measured prompt processing on a 30B model fell from roughly 1,880 tok/s at 8K context to about 425 tok/s at 128K.
Set the context to your actual working size. If your longest realistic prompt is 12K tokens, a 32K window is generous. Reserve the long-context configurations for the specific sessions that need them — this is exactly what the model-switcher script and llama-swap's per-model settings are for.
The GX10 is a 150 mm cube doing a petaflop of sparse arithmetic. Sustained load produces sustained heat, and thermal behaviour shows up as a slow drift in your benchmark numbers rather than an error message.
spark-smi, dgxtop and similar) report unified memory and Grace core behaviour properly; plain free -g plus nvidia-smi also works.Applied in this sequence, each step is measurable before you move on:
The honest summary of this series: the GX10 will not out-generate a big discrete GPU on one conversation, and no amount of tuning changes that. What it will do — hold a model nothing else in its price class can hold, and serve a dozen callers at once while doing it — is a genuinely different capability, and most of the tuning work is about not accidentally configuring it as a slow single-user machine.