The lead article: the component map, measured numbers on this hardware, when retrieval is the wrong answer, and whether to assemble a stack or deploy a finished one.
Retrieval-augmented generation has a simple premise and a great deal of accumulated folklore. The premise: a language model cannot know what is in your files, so find the relevant passages first and put them in the prompt. Everything else is engineering — and most of the quality comes from parts of that engineering people find boring.
This series is opinionated about where the effort goes, because on this hardware the model is rarely the bottleneck. It is nearly always ingestion or retrieval.
documents ──▶ [ parse ] ──▶ [ chunk ] ──▶ [ embed ] ──▶ ┌──────────┐
pdf/docx Docling heading- bge-m3 │ vector │
xlsx/pptx + OCR aware │ store │
md notes └────┬─────┘
│
question ──▶ [ embed ] ──▶ [ hybrid search ] ──▶ [ rerank ] ─┘
dense + sparse cross-encoder
│
▼
[ prompt with sources ] ──▶ LLM ──▶ answer + citationsSix stages, and their contribution to final quality is roughly the inverse of the attention they usually get:
| Stage | Effect on answer quality | Usual effort spent |
|---|---|---|
| Parsing | Enormous. A table mangled into prose cannot be retrieved correctly by anything downstream. | Minimal |
| Chunking | Large. Chunks that split an idea in half retrieve half an idea. | Minimal |
| Embedding model | Moderate, and language-dependent | Large — this is where the arguing happens |
| Hybrid search | Moderate; matters most for names, codes and rare terms | Small |
| Reranking | Large, for very little work | Often skipped entirely |
| Generation model | Smaller than you would think, above a certain size | Large |
If you take one thing from this article: fix parsing and add a reranker before you upgrade the model. A 30B model with good retrieval beats a 120B model with poor retrieval, and it runs three times faster.
A documented GB10 RAG deployment — vLLM serving a 26B mixture-of-experts model with an fp8 KV cache at 65K context, bge-m3 embeddings, a bge-reranker-v2-m3 cross-encoder, Docling for parsing — reports:
| Metric | Measured |
|---|---|
| Time to first token | 183 ms |
| Single-stream generation | 23–24 tok/s |
| Three concurrent requests | ~50 tok/s aggregate |
| Maximum concurrency at 65K context | 45 parallel requests |
| Memory footprint | ~95 GiB (48.5 GiB weights + 41.7 GiB KV cache) |
| Docling parsing a 5-page PDF (warm) | 6.04 s |
Read the last two rows together with the memory budget. The generation model and its cache take about 95 GiB of a roughly 120 GiB usable pool. The embedding model and reranker are 1–2 GB each. It fits — with less headroom than you might have assumed, which is why the context length and the KV cache dtype are the settings to adjust first if you run short.
Retrieval is not free — in setup, in maintenance, or in failure modes. There are cases where the simpler thing is better, and it is worth being honest about them before spending a weekend:
RAG earns its keep on a large, messy, heterogeneous corpus where you do not know in advance which document holds the answer. That is exactly the case you described — mixed PDFs and Office files plus years of personal notes — so this series proceeds.
Two languages in one corpus, with Polish among them, breaks several defaults quietly.
Embedding models are not equally good in every language. An English-first model will embed Polish text, produce plausible vectors, and retrieve worse than you expect — with no error to alert you. The models to consider are the explicitly multilingual ones: bge-m3, MIT-licensed, covering 100+ languages and producing dense, sparse and multi-vector representations from a single pass, and the Qwen3-Embedding family, which leads multilingual retrieval benchmarks in its larger sizes. This series uses bge-m3 as the default because getting dense and sparse retrieval from one model simplifies the hybrid step considerably.
Keyword search assumes English morphology. Polish is heavily inflected: umowa, umowy, umowie, umową, umów are one word. A BM25 index with English stemming treats them as five, so a query using one form misses documents using another. This is why the sparse side of a hybrid system needs language-aware handling, covered in article 3.
Cross-language retrieval needs to be a decision. Should a Polish question surface relevant English documents? Multilingual embeddings make this possible — it is one of their main attractions — but it needs to be intentional, and it needs to be tested, because it also makes spurious cross-language matches possible.
For evaluating Polish retrieval seriously, PL-MTEB is the benchmark that exists — 30 Polish-language tasks across classification, clustering, pair classification, retrieval and semantic similarity, with published results for a wide set of Polish and multilingual models. It is a far better basis for choosing an embedding model for a Polish corpus than an English leaderboard.
Three routes, and the right one depends on what you want to spend your time on.
| Route | What it is | Choose it when |
|---|---|---|
| Assemble | Docling + your chunker + bge-m3 + Qdrant + a reranker + your own answering service, wired by hand | You want to understand and control every stage, and to tune for Polish specifically. This is what the series builds. |
| A finished RAG engine | RAGFlow — a complete system with a web UI, deep document parsing, multilingual OCR and knowledge-base management, no code required. There is a GB10-specific build. | You want it working today and are content with its opinions about chunking and retrieval. |
| A packaged GB10 stack | A one-command private RAG deployment built exclusively for this hardware: vLLM, bge-m3 embeddings, bge-reranker-v2-m3, Docling, Weaviate or Qdrant, RAGFlow, Dify as orchestrator, Open WebUI, MinIO, Postgres, Redis — an interactive wizard and about 25 minutes. | You want the whole thing, correctly configured for unified memory, without making twenty decisions first. |
There is a fourth option worth knowing about for the notes half of your corpus specifically. Graph-based approaches such as LightRAG extract entities and relationships during indexing and retrieve over the resulting graph rather than over isolated chunks. For a personal knowledge base — where the value is often in connections between notes rather than in any single note — this answers questions that vector search structurally cannot, such as "how does this project relate to that one." It is more expensive to index and it is a genuine addition rather than a replacement. Run it alongside vector retrieval over the same corpus rather than choosing between them.
Article 2 handles ingestion: parsing documents that resist parsing, chunking that respects structure, and the metadata that makes citations possible. Article 3 builds hybrid retrieval and reranking, with the Polish-specific adjustments. Article 4 puts an API and a UI in front of it, and — the part most guides omit — an evaluation harness, so that when you change something you can tell whether it helped.