Chatterbox Multilingual v3 for Polish

The harder half: surviving the ARM64 wheel trap, cloning a voice into Polish, tuning prosody, keeping the watermark, and wrapping the result in an OpenAI-compatible server.

Series 2, article 3 of 4 · written 31 July 2026

There is no purpose-built GB10 container for this one. You are installing a Python package on an ARM64 machine with CUDA 13 against a project whose dependency pins assume neither. This article is mostly about doing that without silently ending up on the CPU, because that failure is quiet, common, and costs people entire evenings.

Why this model

Chatterbox Multilingual v3 was released on 10 June 2026: 0.5B parameters, MIT licence, 23+ languages including Polish and Italian, zero-shot voice cloning from around ten seconds of reference audio with no fine-tuning step. Version 3's stated improvements over v2 are exactly the ones that matter for this use case — better speaker similarity when a cloned voice switches language, fewer hallucinations (the repetition and off-prompt drift that plague small TTS models), and more natural conversational delivery.

It also embeds a neural watermark in every output, which survives MP3 compression and is detectable with near-perfect accuracy. Treat that as a feature rather than an annoyance — see the labelling discussion in article 1.

The install, carefully

Chatterbox pins torch==2.6.0, torchaudio==2.6.0 and numpy<2.0.0. On this machine those pins are poison: pip looks for an aarch64 + CUDA-12 wheel for Torch 2.6.0, does not find one, and resolves to the CPU build. Everything installs cleanly. Everything runs. It is roughly thirty times slower than it should be and nothing tells you why.

The fix is to install the package without dependency resolution and then supply the dependencies yourself.

sudo apt install -y python3-venv ffmpeg git
python3 -m venv /opt/spark/tts-pl/venv
source /opt/spark/tts-pl/venv/bin/activate

# 1. Torch FIRST, from NVIDIA's CUDA 13 index, so nothing can downgrade it
pip install --index-url https://download.pytorch.org/whl/cu130 \
    torch torchaudio

# 2. Verify before going further. Do not skip this.
python - <<'PY'
import torch
print("torch:", torch.__version__, "cuda:", torch.version.cuda,
      "available:", torch.cuda.is_available())
print("device:", torch.cuda.get_device_name(0) if torch.cuda.is_available() else "NONE")
assert torch.cuda.is_available(), "CPU fallback — stop and fix this now"
PY

# 3. Chatterbox itself, with its pins ignored
git clone https://github.com/resemble-ai/chatterbox /opt/spark/tts-pl/chatterbox
pip install -e /opt/spark/tts-pl/chatterbox --no-deps

# 4. Its real dependencies, minus torch/torchaudio/numpy pins
pip install transformers librosa soundfile einops \
    "numpy>=2.0" safetensors huggingface_hub resemble-perth \
    fastapi "uvicorn[standard]" pydantic

First synthesis

from chatterbox.mtl_tts import ChatterboxMultilingualTTS
import torchaudio as ta

model = ChatterboxMultilingualTTS.from_pretrained(device="cuda")

wav = model.generate(
    "Pociąg z Krakowa przyjechał z półgodzinnym opóźnieniem.",
    language_id="pl",
    audio_prompt_path="/opt/spark/tts-pl/voices/anna.wav",
    exaggeration=0.5,
    cfg_weight=0.5,
)
ta.save("out-pl.wav", wav, model.sr)

Class and argument names have moved between Chatterbox versions — check the README of the commit you actually cloned. The shape is stable: a model, a language identifier, a reference audio path, and two knobs.

The two knobs

Chatterbox exposes far fewer controls than most TTS systems, which is a mercy. Both of the ones it has matter.

The interaction is the thing to internalise: raising exaggeration speeds delivery up, and lowering cfg_weight slows it back down. For a fast-talking reference voice reading long Polish text, the combination that generally works is exaggeration≈0.4, cfg_weight≈0.4.

Reference audio for Polish specifically

The general guidance from article 2 applies — 10 seconds, clean, one speaker, no clipping, accurate transcript. Two additions for this language:

Record the reference in Polish if you can. Cross-lingual cloning works, and an English reference will produce intelligible Polish. It will also carry an audible English accent, most obviously on ł, rz/ż, the nasal vowels ą and ę, and the palatalised consonants. If the same speaker can give you ten seconds of Polish, the improvement is immediate and large.

Choose reference text with hard sounds in it. A reference full of easy open syllables gives the model less to work with than one containing the clusters Polish actually uses. A sentence with szcz, trz and a nasal vowel is a better ten seconds than a bland one.

Text normalisation, which is on you

This is the single largest quality lever for Polish and it has nothing to do with the model. TTS systems read what you send. Polish inflection means that numbers, dates and abbreviations must be expanded into the correct grammatical case, and no TTS engine will do that for you.

# "o 15:30" → "o piętnastej trzydzieści"     (not "o piętnaście trzydzieści")
# "3 km"    → "trzy kilometry"
# "2. maja" → "drugiego maja"
# "np."     → "na przykład"
# "ul."     → "ulica" / "ulicy" — depends on the case in context

A small dictionary of your own recurring abbreviations covers most of it. For number-to-words with correct Polish declension, use a library rather than regular expressions — this is a genuinely hard problem and someone has already solved it. When the text comes out of an LLM, the cheapest fix of all is to ask the model for it: "write the answer with all numbers, dates and abbreviations expanded into Polish words as they should be spoken."

Wrapping it in an API

To fit the architecture from article 1, this needs the same OpenAI-compatible surface as the Qwen3-TTS containers. About sixty lines:

# /opt/spark/tts-pl/server.py
import io, threading
from pathlib import Path

import torchaudio as ta
from fastapi import FastAPI, HTTPException
from fastapi.responses import Response
from pydantic import BaseModel
from chatterbox.mtl_tts import ChatterboxMultilingualTTS

VOICES = Path("/opt/spark/tts-pl/voices")

app = FastAPI()
model = ChatterboxMultilingualTTS.from_pretrained(device="cuda")
lock = threading.Lock()          # one GPU, one generation at a time


class SpeechRequest(BaseModel):
    model: str = "chatterbox-ml-v3"
    input: str
    voice: str = "anna"
    language: str = "pl"
    response_format: str = "wav"
    exaggeration: float = 0.4
    cfg_weight: float = 0.4


@app.get("/health")
def health():
    return {"status": "ok", "device": "cuda"}


@app.get("/v1/audio/voices")
def voices():
    return {"voices": sorted(p.stem for p in VOICES.glob("*.wav"))}


@app.post("/v1/audio/speech")
def speech(req: SpeechRequest):
    ref = VOICES / f"{req.voice}.wav"
    if not ref.exists():
        raise HTTPException(404, f"unknown voice: {req.voice}")

    with lock:
        wav = model.generate(
            req.input,
            language_id=req.language,
            audio_prompt_path=str(ref),
            exaggeration=req.exaggeration,
            cfg_weight=req.cfg_weight,
        )

    buf = io.BytesIO()
    ta.save(buf, wav, model.sr, format="wav")
    return Response(buf.getvalue(), media_type="audio/wav")

uvicorn server:app --host 0.0.0.0 --port 8030 --app-dir /opt/spark/tts-pl

The lock is not optional. One GPU means one generation at a time; without it, concurrent requests will either produce garbage or exhaust memory. If you need throughput, queue requests rather than parallelising them — TTS runs faster than real time, so a queue drains quickly.

A systemd unit makes it permanent:

# /etc/systemd/system/tts-pl.service
[Unit]
Description=Chatterbox Multilingual TTS (Polish)
After=network-online.target

[Service]
Type=simple
User=YOUR_USER
WorkingDirectory=/opt/spark/tts-pl
Environment=HF_HOME=/mnt/models/hf
ExecStart=/opt/spark/tts-pl/venv/bin/uvicorn server:app --host 0.0.0.0 --port 8030
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

curl -s localhost:8030/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"input":"Dzień dobry. To jest test klonowania głosu.","voice":"anna","language":"pl"}' \
  --output test-pl.wav

The watermark

Every output carries an embedded Perth watermark. It is imperceptible, survives MP3 encoding, and is detectable with near-perfect accuracy. Removing it is technically possible and is a bad idea: from 2 August 2026 the EU AI Act requires machine-readable marking of synthetic audio, and this satisfies that requirement without you writing a line of code. Leave it alone, and keep a note in your pipeline metadata that outputs from this engine are watermarked — because the Qwen3-TTS side is not, and knowing which is which later matters.

Honest expectations

Polish here is good, not indistinguishable. Expect:

• Occasional stress errors on long or unusual words. Polish stress is nearly always penultimate, so errors are audible when they happen. Rephrasing usually fixes an individual case.
• Foreign words read as Polish, or inconsistently. Product names and English technical terms are the usual offenders. Spell them phonetically in the input if it matters.
• A slight accent on cross-lingual clones, discussed above.
• Rare hallucination — a repeated syllable or a trailing invented phrase. v3 reduced this substantially compared with v2 but did not eliminate it. This is exactly what the automated verification loop in article 4 is for: transcribe every output and flag the ones that do not match the input.

If the gap is unacceptable for a specific voice you use constantly, the escalation path is a LoRA fine-tune on a few hours of that speaker's Polish recordings. The box has ample memory for it. Do it only after the zero-shot path has been in real use for a while — most people discover the remaining gap is smaller than they feared, and the ones for whom it isn't will know exactly which voice needs the work.

Sources

• resemble-ai/chatterbox — model family, languages, watermarking, parameters.
• Chatterbox Multilingual v3 release notes, 10 June 2026 — speaker similarity, hallucination and naturalness improvements.
• dgx-spark-setup — the --no-deps install pattern and the CPU-fallback failure mode on GB10.
• NVIDIA forums — torchaudio on ARM64, the same trap in a different project.
• EU AI Act Article 50 — machine-readable marking of synthetic audio from 2 August 2026.