by.waclaw.online / spark / audio / 04

One Voice, Three Languages

A gateway that routes by language, one speaker identity across two engines, long-form chunking, an automated quality loop that listens to its own output, and a browser UI.

Series 2, article 4 of 4 · written 31 July 2026

You now have two speech engines on two ports with two different notions of what a voice is. This article makes that invisible: one endpoint, one voice name, three languages, and a pipeline that catches its own failures before you do.

The persona: what "the same voice" means across engines

The two engines will never produce acoustically identical output. Different models, different training data, different vocoders. What you can achieve is the same person — recognisably one speaker, with consistent character across languages — and that requires being deliberate about the reference material.

Define a persona as a small manifest rather than a single file:

# /opt/spark/voice/personas.yml
anna:
  display: "Anna — narrator"
  consent: "self, recorded 2026-07-12"
  refs:
    en: { engine: qwen,       voice: EN_F_Anna }
    it: { engine: qwen,       voice: EN_F_Anna }
    pl: { engine: chatterbox, voice: anna_pl   }
  defaults:
    exaggeration: 0.4
    cfg_weight: 0.4

lektor:
  display: "Lektor — designed, no real person"
  consent: "synthetic, voice-design seed 'calm male narrator, fifties'"
  refs:
    en: { engine: qwen,       voice: EN_M_Lektor }
    it: { engine: qwen,       voice: EN_M_Lektor }
    pl: { engine: chatterbox, voice: lektor_pl   }

Three practical notes. Record the Polish reference from the same speaker as the English one — that is what keeps the identity coherent across the engine boundary. Record all references in one session, one microphone, one room, one distance; a persona whose references were captured months apart sounds like two people. And keep the consent field populated even when the answer is "it's me" — in six months you will not remember which of a dozen voices came from where, and that is precisely the record you want to have kept.

The gateway

One service on port 8010 that speaks the OpenAI speech API, resolves a persona and language to a backend, normalises text, chunks long input, and writes provenance alongside every output.

# /opt/spark/voice/gateway.py
import io, json, re, subprocess, tempfile, time
from pathlib import Path

import httpx, yaml
from fastapi import FastAPI, HTTPException
from fastapi.responses import Response
from pydantic import BaseModel

PERSONAS = yaml.safe_load(Path("/opt/spark/voice/personas.yml").read_text())
OUT      = Path("/mnt/models/audio-out"); OUT.mkdir(parents=True, exist_ok=True)

BACKENDS = {
    "qwen":       "http://127.0.0.1:8020/v1/audio/speech",
    "chatterbox": "http://127.0.0.1:8030/v1/audio/speech",
}

MAX_CHARS = 350          # chunk boundary; both engines drift on longer input

app = FastAPI(title="spark voice gateway")


class SpeechRequest(BaseModel):
    model: str = "spark-voice"
    input: str
    voice: str = "anna"           # persona name
    language: str = "en"
    response_format: str = "wav"
    keep: bool = True             # write to the archive with provenance


def chunk(text: str) -> list[str]:
    """Split on sentence boundaries, never mid-sentence."""
    parts, buf = [], ""
    for sentence in re.split(r"(?<=[.!?…])\s+", text.strip()):
        if len(buf) + len(sentence) + 1 > MAX_CHARS and buf:
            parts.append(buf.strip()); buf = sentence
        else:
            buf = f"{buf} {sentence}".strip()
    if buf:
        parts.append(buf)
    return parts


def synth_one(backend: str, voice: str, lang: str, text: str,
              defaults: dict) -> bytes:
    payload = {"input": text, "voice": voice, "language": lang,
               "response_format": "wav"}
    if backend == "chatterbox":
        payload |= {k: v for k, v in defaults.items()
                    if k in ("exaggeration", "cfg_weight")}
    r = httpx.post(BACKENDS[backend], json=payload, timeout=300)
    r.raise_for_status()
    return r.content


def concat(chunks: list[bytes], gap_ms: int = 180) -> bytes:
    """Join WAV chunks with a short silence, via ffmpeg."""
    with tempfile.TemporaryDirectory() as d:
        d = Path(d)
        listing = []
        silence = d / "gap.wav"
        subprocess.run(
            ["ffmpeg", "-y", "-f", "lavfi", "-i",
             f"anullsrc=r=24000:cl=mono", "-t", f"{gap_ms/1000}",
             str(silence)], check=True, capture_output=True)
        for i, c in enumerate(chunks):
            p = d / f"{i:04d}.wav"; p.write_bytes(c)
            listing += [f"file '{p}'", f"file '{silence}'"]
        lst = d / "list.txt"; lst.write_text("\n".join(listing[:-1]))
        out = d / "out.wav"
        subprocess.run(["ffmpeg", "-y", "-f", "concat", "-safe", "0",
                        "-i", str(lst), "-c", "copy", str(out)],
                       check=True, capture_output=True)
        return out.read_bytes()


@app.get("/v1/audio/voices")
def voices():
    return {"voices": [
        {"id": k, "display": v["display"],
         "languages": sorted(v["refs"])} for k, v in PERSONAS.items()]}


@app.post("/v1/audio/speech")
def speech(req: SpeechRequest):
    persona = PERSONAS.get(req.voice)
    if not persona:
        raise HTTPException(404, f"unknown persona: {req.voice}")
    ref = persona["refs"].get(req.language)
    if not ref:
        raise HTTPException(
            400, f"persona '{req.voice}' has no voice for '{req.language}'")

    started = time.time()
    pieces = [synth_one(ref["engine"], ref["voice"], req.language, c,
                        persona.get("defaults", {}))
              for c in chunk(req.input)]
    audio = pieces[0] if len(pieces) == 1 else concat(pieces)

    if req.keep:
        stem = OUT / f"{int(started)}-{req.voice}-{req.language}"
        stem.with_suffix(".wav").write_bytes(audio)
        stem.with_suffix(".json").write_text(json.dumps({
            "persona": req.voice, "language": req.language,
            "engine": ref["engine"], "engine_voice": ref["voice"],
            "watermarked": ref["engine"] == "chatterbox",
            "consent": persona.get("consent"),
            "text": req.input,
            "chunks": len(pieces),
            "seconds": round(time.time() - started, 2),
            "generated_at": time.strftime("%Y-%m-%dT%H:%M:%S%z"),
        }, ensure_ascii=False, indent=2))

    return Response(audio, media_type="audio/wav")
curl -s localhost:8010/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"voice":"anna","language":"pl","input":"Dzień dobry. Zaczynamy."}' \
  --output pl.wav

curl -s localhost:8010/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"voice":"anna","language":"it","input":"Buongiorno. Cominciamo."}' \
  --output it.wav

Callers never learn that two engines exist. When Qwen3-TTS adds Polish, or a better Polish model appears, you change one line in personas.yml and nothing downstream notices.

Why chunk at 350 characters. Both engines degrade on long input — drift, dropped clauses, occasional invented endings. Short chunks are stable and concatenate cleanly if you keep the joins on sentence boundaries and insert a short silence. The 180 ms gap is doing real work: without it, joins sound clipped; much longer and the delivery sounds hesitant.

The verification loop

Small TTS models fail rarely but silently — a repeated syllable, a dropped clause, an invented trailing phrase. You will not catch these by listening to everything, and you should not have to. Transcribe the output and compare it with the input.

# /opt/spark/voice/verify.py — synthesise, transcribe, compare, flag
import json, re, sys, unicodedata
from pathlib import Path
import httpx

GATEWAY = "http://127.0.0.1:8010/v1/audio/speech"
ASR     = "http://127.0.0.1:8040/v1/audio/transcriptions"   # Parakeet / WhisperX


def norm(s: str) -> str:
    s = unicodedata.normalize("NFKC", s.lower())
    s = re.sub(r"[^\w\s]", " ", s)
    return " ".join(s.split())


def wer(ref: str, hyp: str) -> float:
    r, h = norm(ref).split(), norm(hyp).split()
    d = [[0] * (len(h) + 1) for _ in range(len(r) + 1)]
    for i in range(len(r) + 1): d[i][0] = i
    for j in range(len(h) + 1): d[0][j] = j
    for i in range(1, len(r) + 1):
        for j in range(1, len(h) + 1):
            d[i][j] = min(d[i-1][j] + 1, d[i][j-1] + 1,
                          d[i-1][j-1] + (r[i-1] != h[j-1]))
    return d[-1][-1] / max(len(r), 1)


def check(text: str, voice: str, lang: str, threshold: float = 0.15):
    audio = httpx.post(GATEWAY, json={"input": text, "voice": voice,
                                      "language": lang}, timeout=300).content
    tr = httpx.post(ASR, files={"file": ("a.wav", audio, "audio/wav")},
                    data={"language": lang}, timeout=300).json()["text"]
    score = wer(text, tr)
    return {"wer": round(score, 3), "ok": score <= threshold,
            "heard": tr, "expected": text, "audio": audio}


if __name__ == "__main__":
    text, voice, lang = sys.argv[1], sys.argv[2], sys.argv[3]
    r = check(text, voice, lang)
    Path("out.wav").write_bytes(r.pop("audio"))
    print(json.dumps(r, ensure_ascii=False, indent=2))

Read the score correctly. A word error rate is measuring two things at once — the synthesiser's mistakes and the recogniser's. A WER of 0.05 on Polish is excellent; 0.10 is normal and mostly the ASR's fault; anything over 0.20 usually means something genuinely went wrong and is worth listening to. Set the threshold from your own baseline rather than from this paragraph: run twenty known-good sentences, look at the distribution, and put the line above it.

Wired into a batch job — narrating a script, generating a library of prompts — this turns "listen to forty minutes of audio" into "listen to the three files it flagged."

The recogniser

For the ASR side, Parakeet TDT 0.6B v3 is the efficient choice on this box: it covers 25 European languages including Polish and Italian, with automatic language identification, at a fraction of Whisper's size and dramatically higher throughput. Where you need word-level timestamps — subtitles, or the lip-sync work in the video series — use WhisperX instead, which adds forced alignment and diarization on top of Whisper. There is a GB10 build of it.

# WhisperX for timings the video series will need
whisperx input.wav --language pl --model large-v3 \
  --output_format srt --highlight_words True

A browser interface

Open WebUI already speaks the OpenAI speech API, so pointing it at the gateway gives you spoken replies from the LLM stack in series 1 with no additional software. In Settings → Audio → Text-to-Speech:

Engine:      OpenAI
Base URL:    http://spark.local:8010/v1
API key:     not-needed
Model:       spark-voice
Voice:       anna

That covers reading answers aloud. For the other job — pasting a paragraph, choosing a language, listening, adjusting — a thirty-line Gradio page is more direct than making Open WebUI do something it was not designed for:

# /opt/spark/voice/ui.py
import gradio as gr, httpx, tempfile

def speak(text, persona, language, exaggeration, cfg):
    r = httpx.post("http://127.0.0.1:8010/v1/audio/speech", timeout=600,
                   json={"input": text, "voice": persona,
                         "language": language})
    r.raise_for_status()
    f = tempfile.NamedTemporaryFile(suffix=".wav", delete=False)
    f.write(r.content); f.close()
    return f.name

personas = [v["id"] for v in
            httpx.get("http://127.0.0.1:8010/v1/audio/voices").json()["voices"]]

gr.Interface(
    fn=speak,
    inputs=[gr.Textbox(lines=8, label="Text"),
            gr.Dropdown(personas, value=personas[0], label="Persona"),
            gr.Radio(["en", "it", "pl"], value="pl", label="Language"),
            gr.Slider(0.0, 1.0, 0.4, label="Exaggeration (Polish only)"),
            gr.Slider(0.0, 1.0, 0.4, label="CFG weight (Polish only)")],
    outputs=gr.Audio(label="Output"),
    title="Spark voice",
).launch(server_name="0.0.0.0", server_port=7861)

What this hands to the video series

The gateway is the audio source for everything in series 3. Lip-synced video is driven by an audio track; a dubbing workflow needs the same speaker saying the same thing in a different language; and word-level timings from WhisperX are what let you cut video to speech rather than the other way round.

Two habits, established now, will save work there:

Sources