Retrieval: Hybrid Search and Reranking

bge-m3 producing dense and sparse representations from one pass, a cross-encoder as the quality step everyone skips, Qdrant as the store, and the morphology problem that makes Polish behave differently.

Series 4, article 3 of 4 · written 31 July 2026

Retrieval is a two-stage problem, and conflating the stages is why so many systems return plausible but wrong passages. Stage one is recall: get the right chunk into a candidate set of fifty, cheaply. Stage two is precision: order those fifty properly, expensively. Vector search is good at the first and mediocre at the second, which is why the reranker matters so much.

The embedding model

For a mixed Polish and English corpus, bge-m3 is the default recommendation and the reason is architectural rather than a leaderboard position: it produces dense, sparse (lexical-weighted) and multi-vector representations from a single forward pass, over 100+ languages, under an MIT licence. Getting both halves of a hybrid search from one model removes an entire class of "the two indexes disagree about tokenisation" problems.

The alternative worth considering is the Qwen3-Embedding family, which leads multilingual retrieval benchmarks in its larger sizes — the 0.6B fits in about 1.5 GB and the 8B is stronger still, and on this machine you can afford the 8B without thinking about it. If you go that route you will need a separate sparse index.

Serving embeddings

Do not load the embedding model inside your ingestion script. Serve it, so the same instance serves ingestion, query time and any other tool, and so its memory is accounted for once.

# A dedicated embedding server, OpenAI-compatible
docker run -d --name embeddings --gpus all -p 8001:8001 \
  -v /mnt/models:/models \
  <your-gb10-vllm-image> \
  vllm serve /models/bge-m3 \
    --task embed --served-model-name bge-m3 \
    --host 0.0.0.0 --port 8001

curl -s localhost:8001/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"bge-m3","input":["umowa najmu lokalu użytkowego"]}' \
  | jq '.data[0].embedding | length'

The infinity server is a good alternative if you want embeddings and reranking from one process; either is fine. Together the embedding model and the reranker are a couple of gigabytes — a rounding error next to the generation model's ~95 GiB.

Why Polish breaks the sparse half

Dense vectors handle morphology reasonably well: the embedding for umowie lands near the embedding for umowa because the model learned they are related. Keyword search does not get that for free.

Polish inflects heavily. Umowa, umowy, umowie, umową, umów are one lemma and five surface forms. A BM25 index built with English stemming — the default in most stacks — treats them as five unrelated tokens, so a query in one form misses documents written in another. English speakers rarely notice this class of bug because English morphology is thin enough that the default mostly works.

Three ways to handle it, in increasing order of effort:

1. Lean on bge-m3's learned sparse weights instead of classical BM25. They come from a multilingual model rather than an English stemmer, so they do not carry the same assumption. This is the low-effort answer and it is usually good enough.
2. Configure the analyser per language. If your store's keyword index supports language-specific analysers, set Polish for Polish documents and English for English ones — which is what the lang field from the previous article was for.
3. Lemmatise at index and query time with a Polish morphological analyser. Most accurate, most work, and worth it only if you have measured that the first two are failing you.

There is also a query-side trick that costs nothing: expand the user's question into two or three paraphrases with the LLM before searching, and union the results. On an inflected language this quietly solves a good deal of the surface-form mismatch, and it helps in English too.

The store

Qdrant is a good default here — one binary, native hybrid support, straightforward filtering on metadata. Weaviate is equally reasonable; the GB10 stacks in circulation use both.

docker run -d --name qdrant -p 6333:6333 \
  -v /mnt/rag/qdrant:/qdrant/storage qdrant/qdrant

# /opt/spark/rag/index.py — build a hybrid collection
import json, httpx
from qdrant_client import QdrantClient, models

EMB = "http://127.0.0.1:8001/v1/embeddings"
qc  = QdrantClient(url="http://127.0.0.1:6333")
COLLECTION = "corpus"

qc.recreate_collection(
    COLLECTION,
    vectors_config={"dense": models.VectorParams(
        size=1024, distance=models.Distance.COSINE)},
    sparse_vectors_config={"sparse": models.SparseVectorParams()},
)

# Metadata fields you will filter on must be indexed explicitly
for field, schema in [("lang", "keyword"), ("doc_type", "keyword"),
                      ("modified", "keyword"), ("source", "keyword")]:
    qc.create_payload_index(COLLECTION, field_name=field,
                            field_schema=schema)


def embed(texts: list[str]) -> list[list[float]]:
    r = httpx.post(EMB, json={"model": "bge-m3", "input": texts}, timeout=120)
    r.raise_for_status()
    return [d["embedding"] for d in r.json()["data"]]


def index(chunks: list[dict], batch: int = 32):
    for i in range(0, len(chunks), batch):
        part = chunks[i:i + batch]
        vecs = embed([c["embed_text"] for c in part])
        qc.upsert(COLLECTION, points=[
            models.PointStruct(
                id=c["id"],
                vector={"dense": v},
                payload={"text": c["text"], **c["meta"]},
            ) for c, v in zip(part, vecs)])
        print(f"indexed {i + len(part)}/{len(chunks)}")


if __name__ == "__main__":
    chunks = [json.loads(l) for l in open("/mnt/corpus/chunks/all.jsonl")]
    index(chunks)

Index the embed_text — the version carrying the heading path — and store the plain text in the payload for display. That distinction, established in the previous article, is what makes fragments retrievable and citations readable at the same time.

Reranking, the step worth its cost

A cross-encoder reads the query and a candidate passage together and scores the pair. That is fundamentally more informative than comparing two independently computed vectors, and it is why reranking typically improves answer quality more than any other single change. It is also why it cannot be used for search itself: scoring every chunk in the corpus against every query is infeasible. Hence the two stages.

bge-reranker-v2-m3 pairs naturally with bge-m3 — same family, same multilingual coverage, Apache-2.0. Retrieve 40–50 candidates, rerank, keep the top 5–8.

# /opt/spark/rag/search.py
import httpx
from qdrant_client import QdrantClient, models

EMB      = "http://127.0.0.1:8001/v1/embeddings"
RERANK   = "http://127.0.0.1:8002/rerank"
qc       = QdrantClient(url="http://127.0.0.1:6333")
CANDIDATES, KEEP = 50, 6


def search(question: str, lang: str | None = None, keep: int = KEEP):
    qv = httpx.post(EMB, json={"model": "bge-m3", "input": [question]},
                    timeout=60).json()["data"][0]["embedding"]

    flt = None
    if lang:
        flt = models.Filter(must=[models.FieldCondition(
            key="lang", match=models.MatchValue(value=lang))])

    hits = qc.search(
        "corpus", query_vector=("dense", qv), limit=CANDIDATES,
        query_filter=flt, with_payload=True)

    if not hits:
        return []

    scored = httpx.post(RERANK, timeout=120, json={
        "model": "bge-reranker-v2-m3",
        "query": question,
        "documents": [h.payload["text"] for h in hits],
    }).json()["results"]

    ranked = sorted(scored, key=lambda r: -r["relevance_score"])[:keep]
    return [{
        "text":   hits[r["index"]].payload["text"],
        "source": hits[r["index"]].payload.get("source"),
        "page":   hits[r["index"]].payload.get("page"),
        "path":   hits[r["index"]].payload.get("heading_path"),
        "score":  round(r["relevance_score"], 4),
    } for r in ranked]

The reranker's absolute scores are also a useful signal in their own right. If the best candidate scores poorly, the honest response is "I could not find this in your documents" rather than an answer built from irrelevant passages. A threshold below which you refuse to answer is one of the cheapest hallucination defences available, and the next article wires it in.

Retrieval settings that matter

Where retrieval fails, and what to do

Diagnose in that order. It is almost always ingestion or chunking, and almost never the embedding model — which is the opposite of where most people look first.

Sources

• BGE / FlagEmbedding — bge-m3's multi-representation design and the bge-reranker-v2-m3 family.
• PL-MTEB and its evaluation code — Polish retrieval evaluation.
• Qdrant documentation — hybrid collections, sparse vectors, payload indexes.
• vLLM pooling and embedding models.
• AGmind — bge-m3 plus bge-reranker-v2-m3 as a working GB10 configuration.