The Deployed Dubbing Pipeline (v2)

Isochrony is the property that dubbed speech occupies the same time as the speech it replaces. Humans dubbing for film obsess over it, because the eye is unforgiving: audio that runs past a closed mouth, or ends while the lips keep moving, reads as broken even to a viewer who can't say why. Most automated dubbing treats isochrony as a post-processing problem to be patched with a time-stretch. This pipeline treats it as a constraint to design around.

Two systems, one problemThis white paper covers the deployed v2 pipeline, which reaches isochrony by prompting Gemini against a phoneme budget. There is also a more advanced v3 research track that reaches it by fine-tuning Llama-3.1-8B — with full training telemetry and eval numbers — written up separately in Teaching an LLM to Fit Its Mouth. Same problem, two architectures.

01Why translation breaks timing

The naive dubbing pipeline is: transcribe → translate → synthesise → paste. It fails at the seam between translate and synthesise. "There are plants in the soil" is six English syllables; a faithful Hindi rendering might be eleven or fifteen. Synthesise that honestly and it overruns the 6.5-second slot the original occupied. The usual rescue is FFmpeg's atempo — speed the audio up to fit — which makes the speaker sound rushed, or slow it to fill, which makes them sound drugged. Either way the artefact is audible, and it compounds over a 400-segment episode.

The core insightYou cannot fix a timing mismatch downstream without damaging the audio. So don't produce one. Move the timing constraint upstream, into the translation step, and generate speech that already fits.

02Constrain upstream, not downstream

The pipeline (v2) runs seven stages, but the whole architecture pivots on stage 3, where translation is made timing-aware before a single sample of audio exists.

Timing enters at translation, not after it. The LLM produces several candidates; a phoneme-budget evaluator picks the one that will fit the slot before any audio is generated.

Upstream of that: Faster-Whisper large-v3 in INT8 transcribes the source into timestamped segments, giving each line a hard duration target. The translation step then treats that target as a budget, not an afterthought.

03The phoneme budget

For each segment the system computes the English syllable count and derives a phoneme target for the slot. It then prompts Gemini with chain-of-thought instructions to produce exactly three translation candidates at different compressions — a direct rendering, a paraphrased one, and a minimal one — and scores each against the phoneme budget, selecting the closest fit.

6.50s

source slot — "There are plants in the soil."

10–14

phoneme/syllable target derived for the slot

92.4%

isochrony score of the selected Hindi candidate

candidates scored per segment before synthesis

The result is a translation chosen partly for meaning and partly for length — the point on the fidelity/brevity trade-off that will actually fit the mouth on screen. This is the step that lets everything downstream avoid time-stretching.

The crack the v3 track closesThe chain-of-thought prompt asks the model to "mentally count the syllables" of each candidate — and language models can't count syllables, because they see tokens, not sounds. Every other piece of v2 (three candidates, an external scorer, a refinement loop) exists to compensate for that one limitation. The v3 fine-tune attacks the limitation directly, teaching the model the phoneme budget through training rather than working around it at inference.

04Preserve the room, not just the voice

A dub that replaces the entire audio track throws away the music, ambience and sound effects that carry half the scene. Stage 1 avoids this with Demucs (htdemucs) source separation, splitting the original into a vocals stem and a background stem. Only the vocals are replaced; the background — the room tone, the score, the traffic — is preserved and remixed under the new dialogue. The vocals stem does double duty: it's what gets transcribed, and it's the reference from which the speaker's voice is cloned. Separating first means the clone hears clean dialogue instead of dialogue-over-music.

05Duration-controlled voice cloning

Synthesis uses IndicF5 (AI4Bharat), a diffusion-transformer TTS built for Indic languages. Two things make it the right choice. First, zero-shot voice cloning: fed a ~12-second clean vocal reference of the original speaker, it generates the translated line in that speaker's voice. Second, and critically, the synthesis is conditioned on the target duration — the TTS is asked to produce ~6.50 seconds of speech, not "whatever length this text wants to be."

Two constraints, one fitThe phoneme budget got the text close to the right length; duration-conditioned TTS lands it exactly. Because both the words and the synthesis respect the slot, the output needs no speed correction — the single biggest source of robotic dubbing is designed out.

06Assemble without stretching

Stage 6 lays each synthesised segment onto its original timeline position by direct overlay — no FFmpeg atempo stretching — with 50 ms crossfades between segments, and mixes the preserved background stem underneath, LUFS-normalised. Stage 7 (FFmpeg) composes the final video and burns in subtitles. Because timing was solved upstream, assembly is a placement problem, not a correction problem.

v1 → v2, the decisions that mattered

Concern	v1	v2 (deployed)
Transcription	openai-whisper base	Faster-Whisper large-v3 (INT8)
Translation	generic prompt	isochrony-aware CoT + phoneme scoring
TTS	Edge-TTS	IndicF5 — duration-controlled
Audio sync	atempo stretch	direct overlay (no stretch)
Voice	generic	zero-shot cloning of the speaker
Background	overwritten	preserved & remixed (Demucs)
Languages	Hindi only	all 11 Indic languages

07SoTA models on a free T4

Running large-v3 Whisper, Demucs and a diffusion-transformer TTS sounds like a cloud-GPU bill. It isn't. The pipeline is designed to run on Kaggle's free dual-T4 accelerator (2×16 GB VRAM, 30 GPU-hours/week), exposed to a local machine via a Cloudflare Tunnel — a "borrowed GPU" pattern that turns a free notebook into a private inference server behind a Streamlit UI. That whole rig is its own write-up: A Headless GPU Lab on Kaggle's Free Tier.

That constraint drove real engineering: INT8 quantisation on the transcription model, careful VRAM staging between stages, and a self-hosted cost of roughly $0.004/minute of processed video on a T4 — the difference between a demo and something you could actually offer as a service.

Key takeaways

Move constraints upstream. Isochrony is unfixable downstream without audible damage, so the translation step is made timing-aware and the mismatch is never created.
Let the LLM optimise for length, not just meaning. Three CoT candidates scored on a phoneme budget pick the fidelity/brevity point that fits the slot.
Separate before you replace. Demucs keeps the original soundstage and gives the voice-clone a clean reference in one move.
Condition synthesis on duration. Duration-controlled IndicF5 lands the exact slot length, eliminating time-stretch artefacts.
Design for the free tier. INT8 + dual-T4 + a tunnel turns SoTA models into a ~$0.004/min service with no cloud bill.

See the interactive walkthrough Read the v3 fine-tune deep-dive

Nirmit K. Tripathii

AI/ML Engineer & Researcher working across speech, LLMs and multimodal systems, with a focus on Indic languages and shipping SoTA models under real-world resource constraints.