The unglamorous stage that decides everything downstream — layout-aware parsing, tables that survive, OCR for Polish scans, structure-respecting chunking, and metadata that makes citation possible.
Nobody enjoys this part, and it is where most of the quality lives. A retrieval system cannot find what parsing destroyed. If a table becomes a run-on paragraph of digits, no embedding model recovers the columns; if a heading is lost, the chunk that follows loses its context permanently.
Docling — an IBM project now hosted by the Linux Foundation — has become the default for local, layout-aware document conversion, and it is a good fit here for a specific reason: it does the right amount of work per page rather than the same work on every page.
Its pipeline runs a layout model first, identifying regions, tables, figures, headings and reading order. It then runs TableFormer on each detected table to recover structure. Only then, and only if the page is a scan with no text layer, does it invoke OCR. On a born-digital PDF, OCR never runs at all and cell text comes from the native text layer — so a table's markdown is TableFormer's structure filled with text no OCR ever touched. That distinction is the difference between a reliable table and a plausible-looking one.
On this hardware, a warm five-page PDF converts in about six seconds.
python3 -m venv /opt/spark/rag/venv
source /opt/spark/rag/venv/bin/activate
# Torch first, from the CUDA 13 index — the hub page's warning applies here too
pip install --index-url https://download.pytorch.org/whl/cu130 torch torchvision
pip install docling docling-core
# Convert one file to check the environment
docling --to md --output /mnt/corpus/out /mnt/corpus/in/umowa.pdfFor a corpus rather than a file, run it as a service so ingestion can be parallel and restartable:
docker run -d --name docling --gpus all -p 5001:5001 \
-v /mnt/corpus:/corpus \
ghcr.io/docling-project/docling-serve-cu130:latestcu124 image will not use the GPU on this machine and may not start. The GB10 RAG stacks in circulation specifically pin the CUDA-13 Docling build; if you are assembling by hand, check the tag rather than assuming.
OCR is where a Polish corpus diverges from an English one. Diacritics — ą, ć, ę, ł, ń, ó, ś, ź, ż — are exactly what a misconfigured OCR engine drops, and the failure is subtle: the text looks nearly right, and every affected word becomes a different token from the one in your query.
ł and ż. If they are absent from a document you know contains them, the language setting did not take effect.Docling handles DOCX, XLSX and PPTX directly. Two notes from experience:
Spreadsheets are usually the wrong thing to embed. A sheet of numbers converted to markdown produces chunks that retrieve badly and answer worse. If the numbers matter, keep the file queryable as data and let the RAG index hold only its prose — the notes, headers and commentary. Semantic search over numeric tables is a reliable source of confident wrong answers.
Markdown notes need no parser and deserve better chunking. An Obsidian-style vault is already structured text; running it through a document parser only loses the structure. Read the files directly, keep the heading hierarchy, and treat wiki-links and tags as metadata — they are a hand-built graph and it would be a waste to discard it.
The instinct is to split every 512 tokens. Do not. Fixed-size chunking cuts through the middle of ideas, and a chunk containing the second half of one argument and the first half of the next retrieves for neither.
The rule that works: one chunk is one idea, and it carries enough context to be understood alone. In practice that means splitting on structure — a heading section, a list, a table — with a modest overlap and, critically, the heading path prepended.
| Content | Strategy | Size |
|---|---|---|
| Prose documents | Heading-aware; split at H2/H3 and subdivide long sections at paragraph boundaries | ~1,500–2,000 characters, ~15% overlap |
| Tables | Never split. One table, one chunk, with its caption and the heading above it | Whatever the table is |
| Markdown notes | One H2/H3 section plus its list items; very short notes stay whole | Whole note if under ~2,000 characters |
| Contracts, structured documents | Split by clause or numbered section — the document's own structure is better than any heuristic | One clause |
| Transcripts | By speaker turn or topic shift, never by fixed length | ~1,500 characters |
Fifteen per cent overlap, not fifty. Large overlaps inflate the index, return near-duplicate results that crowd out genuinely different sources, and mostly compensate for chunking that should have respected structure in the first place.
The single highest-return trick in ingestion: prepend the document title and heading path to each chunk's embedded text. A chunk reading "This does not apply to contracts concluded before that date" is meaningless alone and will never retrieve correctly. The same chunk embedded as:
Umowa najmu — Aneks nr 2 > §4 Okres obowiązywania > Wyłączenia
This does not apply to contracts concluded before that date.is retrievable, and the model receiving it can tell what it is looking at. Embed the contextualised version; keep the original text for display and citation.
# /opt/spark/rag/chunk.py — heading-aware chunking with context prefixes
import re
from dataclasses import dataclass, field, asdict
MAX_CHARS, OVERLAP = 1800, 0.15
@dataclass
class Chunk:
text: str # what gets shown and cited
embed_text: str # what gets embedded (context prefix + text)
meta: dict = field(default_factory=dict)
def split_markdown(md: str, title: str, source: str, lang: str) -> list[Chunk]:
"""Split on headings; subdivide long sections on paragraph boundaries."""
out, path, buf = [], [], []
def flush():
if not buf:
return
body = "\n".join(buf).strip()
if not body:
return
crumbs = " > ".join([title] + path)
for piece in _subdivide(body):
out.append(Chunk(
text=piece,
embed_text=f"{crumbs}\n\n{piece}",
meta={"source": source, "title": title, "lang": lang,
"heading_path": path.copy()},
))
buf.clear()
for line in md.splitlines():
m = re.match(r"^(#{1,6})\s+(.*)", line)
if m:
flush()
level, text = len(m.group(1)), m.group(2).strip()
path[:] = path[: level - 1] + [text]
else:
buf.append(line)
flush()
return out
def _subdivide(body: str) -> list[str]:
if len(body) <= MAX_CHARS:
return [body]
paras, chunks, cur = body.split("\n\n"), [], ""
for p in paras:
if len(cur) + len(p) + 2 > MAX_CHARS and cur:
chunks.append(cur.strip())
tail = cur[-int(MAX_CHARS * OVERLAP):]
cur = tail + "\n\n" + p
else:
cur = f"{cur}\n\n{p}".strip()
if cur.strip():
chunks.append(cur.strip())
return chunksEvery chunk should carry enough to answer "where did this come from" without another lookup, and enough to filter on. At minimum:
{
"source": "/mnt/corpus/in/umowa-najmu-2024.pdf",
"title": "Umowa najmu lokalu",
"heading_path": ["§4 Okres obowiązywania", "Wyłączenia"],
"page": 7,
"lang": "pl",
"doc_type": "contract",
"modified": "2024-11-03",
"ingested": "2026-07-31",
"parser": "docling-2.x",
"ocr": false,
"content_hash": "sha256:…"
}Each field does specific work. page and heading_path make citations verifiable rather than decorative. lang lets you filter or route by language, which matters for a mixed Polish and English corpus. modified lets you prefer recent documents when two versions disagree — extremely common with contracts and notes, and a frequent source of confidently wrong answers. ocr flags text you should trust less. And content_hash is what makes re-ingestion incremental instead of a full rebuild.
You will re-run this. Parsers improve, chunking changes, documents get added. Design for it from the start:
# /mnt/corpus layout that survives a year of changes
in/ # originals, never modified
md/ # docling output, one .md + .json per source
chunks/ # jsonl, one line per chunk, includes config version
state.db # sqlite: content hash → parse status, chunk count, errors
failed.log # what did not parse, and whyWith text in structured chunks and honest metadata attached, the interesting part becomes possible: finding the right ones.