ALL WRITING
Article · Building an Indic Dubbing Pipeline · v2.5 Ship Log

Dubbing v2.5: From a Free T4 to a GPU Job API

v2.5 stops guessing at length and stops ignoring meaning. Then it has to run somewhere. This is how the updated pipeline works end to end, how it deploys today on a free Kaggle T4, and what the move to an async GPU job API has measured so far, with every projection labelled as one.

DeploymentMLOpsIsochronyModalIndic NLP

Version 2 of my dubbing pipeline could translate a line to roughly the right length. It could not tell you whether the shorter line still meant the same thing.

Worse, the ruler it measured "length" with was a vowel-group syllable heuristic that the repo records as roughly 0.43× off the truth, with a different bias for every language. The pipeline was optimising hard against a number that was wrong, and it had no check at all on the thing that mattered more.

v2.5 fixes both. Then it has to run somewhere cheap. This write-up covers the updated code in nirmitktripathii/dubbing_app: what changed, how a single line is chosen, the seven stages from video in to dubbed video out, and the two deployment tracks. One is the free Kaggle T4 setup that runs today. The other is the GPU job API that is being built and measured now.

Scope noteTeaching an LLM to Fit Its Mouth covers the v3 fine-tune research track, and The Deployed Dubbing Pipeline (v2) covers the previous version. This one is the v2.5 code. Numbers are labelled: measured means read from a real run log in the repo; projection means arithmetic on measured parts that has not been run yet.
44/44
validation segments scored for meaning, across all 11 languages
42/44
passed both the meaning and the length gate; 2 reported as misses
0.838
average cross-lingual semantic similarity (IndicSBERT)
0.934
average isochrony score across the 44 segments

01What v2.5 changed

Five changes. In each case v2 was guessing about something it could have measured.

Areav2v2.5
Length rulerVowel-group syllable heuristic (~0.43× off, language-dependent bias)Real phonemes via espeak-ng G2P (phonemes:espeak-ng-1.50) plus IndicNLP normalisation
Meaning checkNone. Any grammatical translation passedIndicSBERT cross-lingual gate: cosine ≥ 0.70 against the English source
Choosing a lineOne generation; take what comes backIterative chain-of-thought loop: 3 candidates per segment, meaning gate first, closest phoneme count wins, up to 3 refinement rounds
When it can't winSilent failureKeeps the best candidate it saw, reports gates_passed=False, and stamps which ruler measured it (espeak-ng or the heuristic fallback)
Non-Devanagari scriptsMiniLM collapsed 8 of 11 languages to a meaningless ~0.99 similarityIndicSBERT discriminates, e.g. 0.72–0.94 on Tamil, Telugu and Odia

The last row is the one I would have missed without a test. The old embedder wasn't wrong about Tamil, Telugu or Odia. It was blind to them: it scored translations in 8 of the 11 languages at about 0.99 whatever they said. A meaning gate built on it would have passed almost anything. A gate that cannot fail is not a gate.

The fourth row is the one I care about most as an engineer. A pipeline that ships a compromised line and says nothing is worse than one that fails loudly, because the only way to find the bad line is to listen to the whole video.

02How a line gets chosen

For every transcribed segment, the pipeline knows how long the original speaker took. That window becomes a phoneme budget, the ideal count. The translator produces three candidates. Only those that clear the meaning gate go on, and of those the one whose phoneme count is closest to the budget wins. Candidates that miss are refined for up to three rounds. Mean it first, then fit it. Every candidate that clears the gate is also ranked by one combined cost, and the loop tracks the best it has ever seen:

cost = 0.60 · (1 − semantic_similarity)  +  0.40 · |phonemes − ideal| / ideal
        └── meaning (IndicSBERT) ──┘        └──── timing fit (espeak-ng) ────┘

gates passed  ⇔  semantic ≥ 0.70  AND  |phonemes − ideal| / ideal ≤ 0.15

The weights lean toward meaning on purpose. The voice model later in the pipeline controls duration natively, so it can absorb a timing miss of a few percent. Nothing after this step can repair a dropped clause.

If nothing clears both gates after the refinement rounds, the loop does not pretend. It keeps the lowest-cost candidate, marks the segment gates_passed=False and records which ruler measured it. That record is what makes the validation readable.

Validation on the real target

I validated v2.5 end to end on the real deployment target, Kaggle (CPU). The set was all 11 languages × 4 sentences = 44 segments. The real espeak-ng ruler was active, not the fallback. IndicSBERT scored all 44, including the 8 non-Devanagari scripts, and all 44 translations were non-empty. Average semantic similarity was 0.838 and average isochrony 0.934.

42 of 44 passed both gates. The other two were reported as too short at the loop's global minimum. That is exactly the kind of visible, per-segment failure the fourth change was built to produce.

03The seven stages

Around that selection loop, the pipeline is seven stages, video in to video out.

#StageWhat it does
1SeparationDemucs (htdemucs) splits the vocals from the music and background, so the background is never touched
2TranscriptionFaster-Whisper (large-v3, INT8 on Kaggle) produces text and timestamps: the windows everything else must hit
3Isochrony translationThe loop above. Gemma handles bulk candidates and Gemini handles refinement, throttled and disk-cached
4Voice referencePicks a clean ~12 s reference clip of the speaker by signal-to-noise ratio
5Duration-controlled TTSIndicF5 speaks each line at its target duration: native voices in Basic mode, voice cloning in Premium
6AssemblyPlaces each clip on the original timeline with no atempo stretching, 50 ms crossfades and a loudness-normalised (LUFS) mix
7CompositionFFmpeg muxes the dubbed audio and the subtitles back into the original video

The README now opens with a playable demo: a 79-second English science clip dubbed into Hindi, in 13 segments. The background is untouched, and the subtitle tracks share identical timing windows. The opening line runs 00:00.27 → 00:02.87 in both. That identical window is the whole project in one line.

04Living on free-tier quotas

Three candidates per segment, plus refinement rounds, across 11 languages is a lot of LLM calls, and the pipeline runs on free API keys. v2.5.1 is the release that stopped rate limits from killing runs:

  • Split routing. The first pass goes to Gemma (gemma-4-31b-it, falling back to gemma-4-26b-a4b-it). Only refinement calls go to Gemini (gemini-3.1-flash-lite, falling back to gemini-3.5-flash-lite). Bulk calls and refinement calls therefore draw on different models' limits.
  • Client-side throttling. Requests are paced per model, at 30 and 15 requests per minute. The client honours the server's 429 retryDelay instead of hammering, and an optional requests-per-day cap is backed by a daily per-model usage counter.
  • A disk candidate cache, keyed by language and source text. A re-run selects from the cache with zero API calls.
  • Parsing without a schema. Gemma has no response_schema, so structured output is parsed from plain text: strip the code fences, then match balanced brackets.

The cache matters more than it looks. The production audit measured translation for the same class of clip at 152 s, 51 s and 40 s on fresh runs, and 16 s with a warm cache. The spread comes from free-tier 503/429 retries and the cold load of IndicSBERT, not from the model thinking harder.

05Deployment today: a free T4 behind a tunnel

The version that runs today costs nothing. kaggle/deploy_dubbing_cloudflare.py is a headless Kaggle kernel, pushed with kaggle kernels push -p . --accelerator NvidiaTeslaT4. The repo calls it the ShadowGPU pattern:

  1. Install dependencies and write the pipeline files into the kernel.
  2. Patch IndicF5's model.py.
  3. Launch the Streamlit app on port 8501.
  4. Open a Cloudflare Quick Tunnel to it.
  5. Broadcast the tunnel's public URL to an ntfy.sh channel.
  6. Listen on that channel for a SHUTDOWN_DUBBING message, a remote kill switch for the session.

API keys come from Kaggle Secrets by name, and nothing is committed. For batch work there is no UI at all. run_headless.py runs stages 1–7 for Kaggle's "Save & Run All", configured by environment variables and using the same stage modules as the app.

The semantic gate runs on CPU on purpose. Kaggle's GPU lottery can hand out a Tesla P100 (sm_60), and the installed torch build has no kernels for it. So IndicSBERT is pinned to CPU with CUDA_VISIBLE_DEVICES="". Losing a little speed is better than losing the run.

The freeze fix

The worst bug in this deployment was not a crash. IndicF5 could wedge inside a CUDA operation while holding Python's GIL. With the GIL held, the whole Streamlit app froze, including the watchdog thread I had written to catch exactly this. A thread cannot rescue the process it lives in.

The fix moved synthesis into a separate OS process (tts_supervisor.py and tts_worker.py). The parent watches a heartbeat. If the heartbeat stalls, the parent SIGKILLs the child and launches a fresh one, which resumes from what is already on disk. A segment that stalls or crashes the worker too many times is declared poison and degrades to silence, so one bad line never aborts a long render.

06Deployment next: an async job API

Status, stated plainlyThis track lives in deploy/, and its own README still calls it a scaffold. It has not been modal deploy-ed as a public service, and the RapidAPI plans are planned, not live. The timings below come from real run logs in the repo. Anything marked projection is arithmetic, not a run.

The honest latency number drove the design. A full fresh run of an 80-second video takes about 7.8 minutes of wall time. Nobody holds an HTTP request open for that, so the product is shaped as an async job API: FastAPI on Modal.

EndpointWhat it does
POST /v1/dubUpload a video with target_lang and mode (basic or vc). An Idempotency-Key header stops a retried upload from starting a second job
GET /v1/dub/{id}Job status with a live stage string, such as "step 6/7 — TTS segment 9/13"
GET /v1/dub/{id}/downloadThe finished video

Before any GPU starts, the API measures the upload with ffprobe. A video over the plan's ceiling (PLAN_MAX_SECONDS) gets a 413. A video whose duration can't be determined gets a 400: rejected, not waived. The same probe_duration() gates the input and meters the output, so what was allowed in and what gets metered can't disagree.

Model weights live on a Modal Volume and download on the first job only. Job files live on a second volume, and job status in a Modal Dict. The GPU is an L4 by default, and two flags (MODAL_WARM, MODAL_TTS_WARM) trade idle cost for no cold start. Secrets are referenced by name from a Modal secret, never written into code.

The A/B that retired the primer

Premium mode keeps the original speaker's voice, and my first approach to that failed in an instructive way. IndicF5 can be conditioned on a reference clip, so I conditioned it on the English speaker's audio and text, plus a short Hindi primer ("नमस्ते।") that was sliced off after synthesis. In an A/B against the Basic run, Basic was clean and the primer run was garbled.

Timing was not the problem. Every segment drifted under 500 ms, and isochrony averaged 0.94. The cause was cross-lingual conditioning, which destabilised IndicF5's flow-matching alignment across the whole utterance. The onsets gave it away. Basic segments started with 175–450 ms of leading silence, the primer run with roughly none, and the slicer's cut points were scattered anywhere from 0.31 s to 1.57 s.

So the primer is retired, and the fix separates content from timbre. First, synthesise clean native Hindi. Then a new Step 6.5 converts the voice with kNN-VC: zero-shot voice conversion over WavLM features, with no per-speaker training. It preserves duration, so it cannot undo the timing work. The API rejects the cross-lingual mode outright.

07Where the time actually goes

The production audit, a draft on the feat/modal-deploy branch, parsed a real run log for a full fresh 80-second run at nfe=32:

StageSecondsShare
Extract1<1%
Separation (Demucs)123%
Transcription (Whisper medium)163%
Translation15232%
Voice reference2<1%
TTS (IndicF5): 28 s model load + 205 s synthesis, 13 segments at a mean of 15.8 s24051%
Assembly4510%
Total wall468100%

Measured. Source: PRODUCTION_OPTIMIZATION_AUDIT.md in the repo.

Reading that table as a bill instead of a profile changed the plan. The audit draws three conclusions from it:

  • The orchestrator rents a GPU to make phone calls and run FFmpeg. The job runs inside one GPU function, so the L4 is held and billed per second through the 152 s translation call and the 45 s FFmpeg step. 200 of 468 s (43%) of the billed GPU wall is non-GPU work (measured).
  • Fan-out buys latency, not savings. Splitting TTS across containers finishes sooner, but every extra container loads the model again, so total GPU-seconds go up, not down.
  • After fan-out, translation is the critical path, and it should never touch a GPU.

Two fixes from the audit have landed, both verified on CPU by tools/test_merge_and_cache.py. Subtitles are soft-muxed and the video stream is copied rather than re-encoded. The translation cache now persists on the Modal Volume. The rest is still a backlog, and it is labelled that way in the repo: moving the orchestrator off the GPU, stopping the retry storm, stage checkpoints and content-hash idempotency.

TTS fan-out

TTS is the biggest GPU stage, and its segments are independent. tts_fanout.py shards them across TTS containers. Each shard runs the same synthesis function on its slice of global segment indices, so nothing is renumbered, and the orchestrator is the single writer that merges the results. It maps over shards, not segments, so the model loads once per shard rather than once per line.

Expected wall time is model_load + ceil(n / shards) × per_segment. For the 13-segment video:

ShardsColdWarm
1~275 s~235 s
4~112 s~72 s
8~76 s~36 s
13~58 s~18 s

Projection, not measurement. It is computed from the measured serial TTS time of ~235 s for 13 segments at ~18 s each.

One measured data point exists for the fan-out path. A 25-second clip ran end to end on an L4 in 275 s, with fan-out TTS at 82 s across roughly 3–4 containers, dominated by cold model load. The target for a warm 1-minute clip is about 60–90 seconds end to end. That is a projection until it runs.

08What's next

  • Get the orchestrator off the GPU. It is the largest avoidable cost in the audit.
  • Move translation off the free Gemini key, most likely onto IndicTrans2, which also ends the retry storm.
  • Check commercial licensing for IndicF5 and kNN-VC before anything is sold.
  • Store artifacts in S3/R2 instead of the container, and master the final mix to EBU R128 loudness.

The transferable lesson

  • A gate that can't fail isn't a gate. Test your meaning check on the scripts you actually ship.
  • Fail loudly per item. gates_passed=False on one segment beats a silent compromise spread across a video.
  • A thread can't rescue the process it lives in. Put GPU work you may have to kill in its own process.
  • Read a profile as a bill. 43% of the GPU time was the GPU waiting on something else.
  • Label every number as measured or projected, and keep the two apart.
View the repo The v2 pipeline
Nirmit K. Tripathii
Nirmit K. Tripathii — AI/ML Engineer & Football Data Scientist
Three years shipping NLP, agentic AI and multimodal systems. I write up real projects — including the parts that failed. Available now for part-time, consulting and research collaboration (remote), and open to full-time roles from September 2026. If you're working on speech, multilingual NLP, or sports analytics, I'd like to hear about it.