Talking Heads and Dubbing

Driving video from the voices you cloned in series 2 — LipDub for re-voicing footage, InfiniteTalk for long-form presenters — and wrapping ComfyUI in a job API so your own code can call it.

Series 3, article 4 of 4 · written 31 July 2026

This is where the three series meet. An LLM writes the script, the speech gateway renders it in Polish or Italian in a voice you chose, and a video model makes a person say it. Every step runs on the box, which is the point.

Two different jobs

People say "lip sync" for two problems that need different tools.

The important distinction from older lip-sync tools: neither of these masks and repaints the mouth region. They regenerate facial movement conditioned on the audio, so jaw, cheeks and expression move together. That is why they hold up at higher resolution where the mask-and-repaint generation of tools visibly did not.

The audio comes first

Everything downstream is conditioned on the audio track. Regenerating video because you changed the voice costs tens of minutes; regenerating the voice costs seconds. So lock the audio completely before spending a single GPU-minute on frames.

# 1. Script → speech, via the gateway from series 2
curl -s http://127.0.0.1:8010/v1/audio/speech \
  -H 'Content-Type: application/json' \
  -d '{"voice":"anna","language":"pl",
       "input":"Dzień dobry. W tym odcinku pokażę, jak działa cały pipeline."}' \
  --output vo.wav

# 2. Normalise for the video model: mono, 16 kHz, even loudness, no silence
#    at the head (a leading gap makes the first mouth movement look late)
ffmpeg -i vo.wav -ac 1 -ar 16000 \
  -af "silenceremove=start_periods=1:start_threshold=-50dB,loudnorm=I=-18:TP=-2" \
  vo16.wav

# 3. Word-level timings, if you want to cut to the speech
whisperx vo16.wav --language pl --model large-v3 \
  --output_format json --highlight_words True

Three properties matter to the video models: mono, consistently loud (these models take amplitude as an intensity cue, so a quiet passage produces underanimated mouth movement), and clean — no music bed, no ambience. Add atmosphere in the final mux, never in the driving track.

Dubbing existing footage: LTX-2.3 LipDub

LipDub is delivered as an IC-LoRA on top of LTX-2.3 and runs as a two-stage workflow. The Lightricks ComfyUI node pack ships it as a two-stage distilled example graph; load that rather than building it yourself, and make sure you are using the local open-weights variant rather than an example graph that routes to a hosted API key node.

The two stages exist for a reason worth understanding:

1. Stage 1 generates the dubbed video at reduced resolution, with the source video frames and the driving audio both supplied as conditioning. The IC-LoRA is what aligns mouth movement to the speech while the rest of the frame follows the original.
2. Upsample. The video latent is scaled up spatially while the audio latent is frozen — so the timing established in stage 1 is preserved exactly.
3. Stage 2 re-generates at full resolution from the upscaled latent, restoring detail.

Practically: stage 1 is where you check whether the sync works, and it is cheap. Run it alone, watch it, and only then let stage 2 run. Discovering a timing problem after the full-resolution pass is the expensive way to learn it.

What to feed it:

• Source footage with a clearly visible, reasonably front-facing head. Extreme angles, heavy occlusion (a hand across the mouth) and very small faces in frame all degrade results sharply.
• Audio of similar duration to the clip. Polish and Italian renderings of the same English sentence differ in length — often considerably. Either generate the target-language audio to fit, or accept that you will be cutting the source. This is the single most common frustration in cross-language dubbing and it is a scripting problem, not a model problem: write the target-language script to the available time.
• A short clip. Dub in segments at natural pauses and join them. Long single passes drift.

Presenters from a still: InfiniteTalk

When there is no source footage — just a portrait and a script — InfiniteTalk is the long-form option. It is a 14B audio-driven image-to-video model built for continuous, multi-segment speech: rather than only moving lips, it aligns head movement, posture and expression to the audio, and it is architecturally designed for arbitrarily long output with identity held stable throughout.

Reported requirements put the full pipeline around 18–22 GB at peak, with fp8 quantized weights cutting that substantially. That is unremarkable on this machine — the constraint on a 24 GB card, which forces most users to 480p, simply does not apply to you. You can run 720p because you have the memory; it will cost time, not stability.

Getting good results:

• Portrait selection matters more than settings. Front-facing, evenly lit, neutral expression, head and shoulders, nothing crossing the face. A dramatic portrait with strong side-lighting animates into something uncanny.
• Neutral start. A portrait with a fixed broad smile animates into a person who cannot stop smiling, because the model treats the expression as identity.
• Generate in segments anyway. Even with long-form support, cutting between segments at natural pauses gives you something that looks edited rather than something that looks like an unbroken take from a machine — which is both more watchable and more forgiving of drift.
• Resolution is a time dial. Draft at 480p; render the keeper at 720p.

Wan 2.2's S2V-14B is the alternative when you want prompt-guided cinematic motion around the speaker rather than a straightforward presenter shot — a camera move, a scene, an environment. It is heavier and less specialised for long-form.

Wrapping ComfyUI in a job API

ComfyUI is a queue, not a request-response service: you submit a graph, it returns a job ID, and you collect output later. That is exactly right for work that takes minutes, and exactly wrong as an interface for your own code. A small wrapper turns it into something you can call.

# /opt/spark/video/jobapi.py
import json, uuid, copy, urllib.parse
from pathlib import Path

import httpx
from fastapi import FastAPI, HTTPException, UploadFile, File, Form
from fastapi.responses import FileResponse

COMFY     = "http://127.0.0.1:8188"
GRAPHS    = Path("/opt/spark/video/graphs")     # 'Save (API format)' exports
INPUT_DIR = Path("/home/YOUR_USER/ComfyUI/input")
CLIENT_ID = str(uuid.uuid4())

app = FastAPI(title="spark video jobs")


def submit(graph: dict) -> str:
    r = httpx.post(f"{COMFY}/prompt",
                   json={"prompt": graph, "client_id": CLIENT_ID}, timeout=60)
    r.raise_for_status()
    return r.json()["prompt_id"]


@app.post("/v1/video/lipdub")
async def lipdub(audio: UploadFile = File(...),
                 source: UploadFile = File(...),
                 stage2: bool = Form(True)):
    """Re-voice existing footage. Returns a job id immediately."""
    a = INPUT_DIR / f"{uuid.uuid4().hex}.wav"; a.write_bytes(await audio.read())
    v = INPUT_DIR / f"{uuid.uuid4().hex}.mp4"; v.write_bytes(await source.read())

    g = copy.deepcopy(json.loads(
        (GRAPHS / "ltx2_lipdub_two_stage.json").read_text()))
    # Node ids come from your own exported graph — read it once and fix these.
    g["10"]["inputs"]["audio"] = a.name
    g["11"]["inputs"]["video"] = v.name
    if not stage2:
        g["40"]["inputs"]["enabled"] = False

    return {"job": submit(g)}


@app.post("/v1/video/presenter")
async def presenter(audio: UploadFile = File(...),
                    portrait: UploadFile = File(...),
                    width: int = Form(832), height: int = Form(480)):
    """Animate a still portrait from an audio track."""
    a = INPUT_DIR / f"{uuid.uuid4().hex}.wav"; a.write_bytes(await audio.read())
    p = INPUT_DIR / f"{uuid.uuid4().hex}.png"; p.write_bytes(await portrait.read())

    g = copy.deepcopy(json.loads((GRAPHS / "infinitetalk.json").read_text()))
    g["10"]["inputs"]["audio"] = a.name
    g["12"]["inputs"]["image"] = p.name
    g["20"]["inputs"]["width"] = width
    g["20"]["inputs"]["height"] = height

    return {"job": submit(g)}


@app.get("/v1/video/jobs/{job_id}")
def status(job_id: str):
    hist = httpx.get(f"{COMFY}/history/{job_id}", timeout=30).json()
    if job_id not in hist:
        q = httpx.get(f"{COMFY}/queue", timeout=30).json()
        running = any(i[1] == job_id for i in q.get("queue_running", []))
        return {"job": job_id, "state": "running" if running else "queued"}

    outputs, files = hist[job_id]["outputs"], []
    for node in outputs.values():
        for kind in ("gifs", "videos", "images", "audio"):
            for f in node.get(kind, []):
                files.append(f)
    return {"job": job_id, "state": "done",
            "files": [f["filename"] for f in files]}


@app.get("/v1/video/files/{filename}")
def fetch(filename: str, subfolder: str = "", type: str = "output"):
    q = urllib.parse.urlencode(
        {"filename": filename, "subfolder": subfolder, "type": type})
    r = httpx.get(f"{COMFY}/view?{q}", timeout=300)
    if r.status_code != 200:
        raise HTTPException(404, "not found")
    out = Path("/tmp") / filename
    out.write_bytes(r.content)
    return FileResponse(out)

uvicorn jobapi:app --host 0.0.0.0 --port 8090 --app-dir /opt/spark/video

Called end to end, the whole stack becomes four HTTP requests:

# 1. words (series 1) → 2. voice (series 2) → 3. video → 4. collect
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"daily","messages":[{"role":"user",
       "content":"Napisz 30-sekundowy skrypt wideo o tym pipeline."}]}' \
  | jq -r '.choices[0].message.content' > script.txt

curl -s localhost:8010/v1/audio/speech -H 'Content-Type: application/json' \
  -d "$(jq -Rs '{voice:"anna",language:"pl",input:.}' script.txt)" \
  --output vo.wav

JOB=$(curl -s localhost:8090/v1/video/presenter \
        -F audio=@vo.wav -F portrait=@anna.png | jq -r .job)

until [ "$(curl -s localhost:8090/v1/video/jobs/$JOB | jq -r .state)" = done ]; do
  sleep 20
done
curl -s "localhost:8090/v1/video/files/$(curl -s \
  localhost:8090/v1/video/jobs/$JOB | jq -r '.files[0]')" --output out.mp4

Finishing and labelling

The last mux is where the pieces come together and where labelling belongs.

# Voice-over over generated ambience, plus provenance metadata in the file
ffmpeg -i raw.mp4 -i vo.wav -filter_complex \
  "[0:a]volume=0.3[amb];[1:a]volume=1[vo];[amb][vo]amix=inputs=2:duration=first[a]" \
  -map 0:v -map "[a]" -c:v libx264 -crf 18 -c:a aac \
  -metadata comment="AI-generated video and synthetic voice. \
Video: LTX-2.3 / InfiniteTalk. Voice: cloned persona 'anna' (pl), consented. \
Generated 2026-07-31 on local hardware." \
  final.mp4

# A visible label, when the clip is going anywhere public
ffmpeg -i final.mp4 -vf "drawtext=text='AI-generated':x=w-tw-24:y=h-th-24:\
fontsize=22:fontcolor=white@0.75:box=1:boxcolor=black@0.35:boxborderw=8" \
  -c:a copy final-labelled.mp4

From 2 August 2026, EU rules require AI-generated or manipulated audio and video to be machine-readably marked and disclosed, with a reinforced obligation where the content resembles a real person and could be taken as authentic — which is precisely what a cloned-voice talking head is. Metadata is not by itself sufficient marking under every reading, and a visible label is not machine-readable; doing both, consistently, is the defensible position for anything that leaves your machine. For material that stays on it, this is a good habit rather than an obligation.

A quality checklist

When a result is not convincing, it is almost always one of these, in roughly this order of likelihood:

1. The driving audio is quiet, uneven, or has music under it. Fix the audio first, always.
2. The portrait or source frame is poor — angled, oddly lit, occluded, or carrying a fixed expression.
3. The segment is too long. Cut at pauses; join afterwards.
4. The audio and the source clip are different lengths. Write the target-language script to fit the shot.
5. Resolution too low for the framing. A close-up needs more pixels than a mid shot to survive a mouth being regenerated.
6. Only then: model settings.

And the honest summary of this series: the GX10 will not give you a real-time video studio, and nothing in it pretends otherwise. What it gives you is a machine that holds a 22-billion-parameter audio-video model without complaint, runs a full text-to-speech-to-video pipeline in three languages with no data leaving the house, and turns generation into a queue you submit to rather than a thing you watch. For anyone who has tried to fit this workload onto a consumer GPU, that is a different category of tool.

Sources

• LTX documentation — LipDub (IC-LoRA) and ComfyUI-LTXVideo, which ships the two-stage distilled graph.
• InfiniteTalk and the paper — long-form audio-driven video, identity preservation.
• Wan 2.2 S2V — speech-to-video as the alternative pipeline.
• whisperx-gb10 — word-level timings on this hardware.
• ComfyUI documentation — the /prompt, /history, /queue and /view endpoints used by the job wrapper.
• EU AI Act Article 50 — marking and disclosure obligations from 2 August 2026.