Isochrony-Aware Video Dubbing and Zero-Shot Voice Cloning
Select any phase of the pipeline below to see how our engineering system processes the media files in real-time:
First, FFmpeg extracts raw 16kHz mono audio. Then, **Demucs (htdemucs)** separates it into vocal and background stems, allowing us to clone clean dialogue while retaining the original music/SFX track.
We execute **Faster-Whisper (large-v3)** in INT8 quantization on the GPU to generate high-fidelity, timestamped English transcript segments.
We calculate the English syllable count and translate via **Gemini 3.1 Flash-Lite**. Chain-of-Thought (CoT) instructions force the LLM to generate exactly 3 candidates (Direct, Paraphrased, and Minimal), and we score them using a phoneme-budget evaluator to pick the best fit.
Using **IndicF5 (AI4Bharat)**, we feed a 12-second vocal clip from the speaker to extract speaker identity, and generate Hindi speech. The TTS is conditioned on the target duration to prevent downstream speed-stretching.
We assemble the individual synthesized speech segments into a 455.92s timeline, overlay the original background track at 35% volume to preserve the ambient sound, and run FFmpeg to burn in the Nirmala UI Hindi subtitles.