An OpenAI-compatible endpoint that answers with sources, a browser interface, memory orchestration alongside the other workloads, and an evaluation harness so changes can be shown to be improvements.
Retrieval gives you six good passages. This article turns them into an answer you can trust, puts that behind the same OpenAI-compatible interface as everything else on the box, and — the part most guides omit — gives you a way to tell whether a change made things better or merely different.
Most hallucination in a RAG system comes from a prompt that leaves the model no acceptable way to say "it isn't here." Give it one explicitly, and make citation mechanical rather than aspirational.
SYSTEM = """You answer questions using only the numbered sources provided.
Rules:
- Use only information in the sources. Do not use prior knowledge.
- Cite the source number in square brackets after each claim: [2].
- If the sources do not contain the answer, say exactly:
"I could not find this in your documents." Do not guess.
- If sources conflict, say so and prefer the one with the later date.
- Answer in the language of the question.
- Be concise. Do not restate the question."""
def build_context(passages: list[dict]) -> str:
out = []
for i, p in enumerate(passages, 1):
path = " > ".join(p.get("path") or [])
head = f"[{i}] {p['source']}"
if p.get("page"):
head += f", p.{p['page']}"
if path:
head += f" — {path}"
if p.get("modified"):
head += f" (modified {p['modified']})"
out.append(f"{head}\n{p['text']}")
return "\n\n---\n\n".join(out)Three details carry most of the weight. The exact refusal string — a fixed phrase is detectable in logs and in evaluation, so you can measure how often the system declines. Numbered sources make citations checkable: a claim tagged [2] can be verified against passage 2 mechanically. "Answer in the language of the question" matters concretely in a Polish and English corpus, where the passages may not be in the language the person asked in.
# /opt/spark/rag/api.py
import time, uuid
from typing import Optional
import httpx
from fastapi import FastAPI
from pydantic import BaseModel
from search import search # from article 3
LLM = "http://127.0.0.1:8000/v1/chat/completions"
MIN_SCORE = 0.15 # calibrate this on your own corpus
app = FastAPI(title="spark rag")
class AskRequest(BaseModel):
question: str
lang: Optional[str] = None # filter the corpus, not the answer
keep: int = 6
model: str = "daily"
@app.post("/v1/rag/ask")
def ask(req: AskRequest):
t0 = time.time()
passages = search(req.question, lang=req.lang, keep=req.keep)
if not passages or passages[0]["score"] < MIN_SCORE:
return {"answer": "I could not find this in your documents.",
"sources": [], "refused": True,
"best_score": passages[0]["score"] if passages else None}
body = {
"model": req.model,
"temperature": 0.1,
"messages": [
{"role": "system", "content": SYSTEM},
{"role": "user",
"content": f"Sources:\n\n{build_context(passages)}\n\n"
f"Question: {req.question}"},
],
}
r = httpx.post(LLM, json=body, timeout=300)
r.raise_for_status()
answer = r.json()["choices"][0]["message"]["content"]
return {
"answer": answer,
"sources": [{"n": i, "source": p["source"], "page": p.get("page"),
"path": p.get("path"), "score": p["score"]}
for i, p in enumerate(passages, 1)],
"refused": answer.strip().startswith("I could not find"),
"seconds": round(time.time() - t0, 2),
}
@app.post("/v1/chat/completions")
def openai_shim(payload: dict):
"""Minimal OpenAI-compatible surface so existing clients work unchanged."""
question = payload["messages"][-1]["content"]
result = ask(AskRequest(question=question))
cites = "\n\n" + "\n".join(
f"[{s['n']}] {s['source']}" + (f", p.{s['page']}" if s['page'] else "")
for s in result["sources"]) if result["sources"] else ""
return {
"id": f"chatcmpl-{uuid.uuid4().hex[:12]}",
"object": "chat.completion",
"created": int(time.time()),
"model": "spark-rag",
"choices": [{"index": 0, "finish_reason": "stop",
"message": {"role": "assistant",
"content": result["answer"] + cites}}],
}The second endpoint is what makes this usable from software you did not write. Point Open WebUI, an editor plugin or any OpenAI client at http://spark.local:8010/v1 with model spark-rag, and it behaves like a model that happens to know your documents.
Temperature 0.1, not 0.7. This is an extraction task, not a creative one. High temperature here produces fluent paraphrase that drifts from what the sources actually say — which is the worst possible failure mode, because it reads well.
Open WebUI, already running from series 1, needs one addition:
Settings → Connections → OpenAI API
Base URL: http://spark.local:8010/v1
API key: not-neededspark-rag then appears in the model dropdown next to your ordinary models, and switching between "answer from your documents" and "answer generally" is a dropdown rather than a different application. That is a better arrangement than it sounds: the two modes fail in different ways, and being able to ask the same question both ways is a fast way to spot when retrieval is letting you down.
Open WebUI's own built-in knowledge feature still has a place — for a handful of files you want to ask about once, uploading them there is faster than running ingestion. Use both; they solve different problems.
RAG holds three models: the generation model at roughly 95 GiB with its cache, the embedding model at about 1.5 GB, and the reranker at a similar size. Total near 100 GiB of a roughly 120 GiB pool. That leaves no room for a 42 GB video transformer, and the failure when you try is an OOM at an inconvenient moment rather than a polite refusal.
Make the mode switch explicit:
#!/usr/bin/env bash
# /usr/local/bin/spark-mode — one large workload at a time
set -euo pipefail
RAG="vllm embeddings reranker qdrant docling"
VIDEO="comfyui"
AUDIO="qwen3-tts-clone tts-pl" # small; can coexist with either
stop_all() {
for s in $RAG $VIDEO; do
systemctl is-active --quiet "$s" && sudo systemctl stop "$s" || true
done
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null
}
case "${1:-status}" in
rag) stop_all; for s in $RAG $AUDIO; do sudo systemctl start "$s"; done ;;
video) stop_all; for s in $VIDEO $AUDIO; do sudo systemctl start "$s"; done ;;
off) stop_all ;;
status) systemctl is-active $RAG $VIDEO $AUDIO 2>/dev/null | paste -d' ' \
<(echo $RAG $VIDEO $AUDIO | tr ' ' '\n') - ;;
esac
free -g | head -2The speech services are small enough to stay resident in either mode, which is what makes the cross-series pipelines from the video series practical. The cache drop between modes is not decoration — without it the previous workload's pages are still held and the next load hits an avoidable wall.
Every change you make to this pipeline will feel like an improvement. Some will be. Without measurement you cannot tell which, and RAG systems are unusually good at feeling better while getting worse — a change that makes answers more fluent often makes them less faithful.
You do not need a research harness. Thirty questions from your own corpus, with the document that should answer each, is enough to catch every regression that matters.
# /opt/spark/rag/eval/questions.jsonl
{"q": "Jaki jest okres wypowiedzenia w umowie najmu z 2024?", "expect_source": "umowa-najmu-2024.pdf", "expect_contains": ["trzy miesiące"], "lang": "pl"}
{"q": "What did I decide about the storage layout for models?", "expect_source": "notes/2026-04-hardware.md", "expect_contains": ["external", "M.2"], "lang": "en"}
{"q": "Ile wynosi kaucja?", "expect_source": "umowa-najmu-2024.pdf", "expect_contains": [], "lang": "pl"}
{"q": "What is the capital of Peru?", "expect_source": null, "expect_contains": [], "lang": "en"}That last row is doing important work. A question your corpus cannot answer should be refused; a system that answers it from general knowledge has quietly stopped being grounded, and you want that failure to show up in a number.
# /opt/spark/rag/eval/run.py
import json, statistics, sys
import httpx
API = "http://127.0.0.1:8010/v1/rag/ask"
CASES = [json.loads(l) for l in open("questions.jsonl")]
hits, faithful, refusals_ok, times = [], [], [], []
for c in CASES:
r = httpx.post(API, json={"question": c["q"], "lang": c.get("lang")},
timeout=300).json()
times.append(r.get("seconds", 0))
srcs = " ".join(s["source"] or "" for s in r["sources"])
ans = r["answer"].lower()
if c["expect_source"] is None:
ok = r["refused"]
refusals_ok.append(ok)
mark = "REFUSED" if ok else "ANSWERED ANYWAY"
else:
retrieved = c["expect_source"] in srcs
hits.append(retrieved)
contains = all(t.lower() in ans for t in c["expect_contains"])
faithful.append(contains)
mark = f"retrieved={retrieved} contains={contains}"
print(f"{mark:34} {c['q'][:52]}")
def pct(xs): return f"{100 * sum(xs) / len(xs):.0f}%" if xs else "n/a"
print(f"\nretrieval hit rate : {pct(hits)}")
print(f"answer contains : {pct(faithful)}")
print(f"correct refusals : {pct(refusals_ok)}")
print(f"median latency : {statistics.median(times):.1f}s")Run it before and after every change — a new embedding model, a different chunk size, a reranker threshold, a bigger generation model. Four numbers, thirty seconds of reading, and you know whether to keep the change.
Two habits make it more useful over time. Add a case every time the system fails you in real use; within a couple of months the suite reflects your actual corpus rather than what you imagined it contained. And track the numbers separately by language — a change that improves English retrieval while degrading Polish is entirely possible, and an aggregate figure hides it.
Across the four series, the box now runs a single coherent system on one machine on your own network:
| Endpoint | What it does |
|---|---|
:8000/v1 | LLM — chat and completion, concurrent callers |
:8001/v1 | Embeddings |
:8010/v1/rag/ask | Answers grounded in your documents, with citations |
:8010/v1/audio/speech | Speech in English, Italian and Polish, in a voice you chose |
:8090/v1/video/* | Video generation and lip sync as submitted jobs |
:3000 / :8188 | Open WebUI and ComfyUI for the times you want to work by hand |
Nothing in that list requires an internet connection once the weights are downloaded, and nothing sends your documents, your voice or your face to anyone. That, rather than any individual benchmark, is what the machine is for.