Teaching an LLM to Fit Its Mouth

The Hindi dub was two seconds behind by the end of the first paragraph. By the end of the clip it was four.

Nothing was broken. The transcription was accurate, the translation was good, the text-to-speech was clean. The problem was arithmetic.

Here is a row from my own validation set. The English "Milk is rich in calcium." — five short words — becomes:

दूध में कैल्शियम की भरपूर मात्रा मौजूद होती है.

The phonemizer scores that at 39 phonemes. You can see the expansion without counting anything. Do that forty times in a five-minute video and the voice track has walked off the end of the picture.

This is the central problem in automated dubbing. It has an obvious fix that everybody reaches for first — I reached for it too — and a better fix that the frontier labs have now converged on. This article is about a third place to attack it, upstream of both, that I think is being left on the table.

The short version: I moved the constraint into the translation model, which now gets told how long its output is allowed to be, in phonemes, before it writes a single word. That took length error from 49% down to 10% — and, unexpectedly, made the translations better rather than worse.

01Three places to fix it

Making the dub take exactly as long as the original — the literature calls this isochrony — can be enforced at exactly three points in a pipeline. Which one you pick determines what your output sounds like.

Stage 1 — After the audio exists (time-stretching)

If the audio is too long, shorten the audio. Every dubbing pipeline starts here, and it works well enough to demo. My v1 did exactly this: after generating each TTS chunk, it compared the chunk's actual duration against the window the original speaker had used, then squeezed — time-stretch it with Librosa (preserving pitch) if too long, pad with silence via Pydub if too short. The docstring I left on that module is blunt about what it did:

OLD: Generated audio might be wrong length → FFmpeg atempo stretches it.

Pitch-preserving time-stretch is genuinely clever signal processing, and for a 5% squeeze it's inaudible. The trouble is that the errors here aren't 5%. Indic translations of English run routinely 20–40% longer, and sometimes worse. Compress speech by 35% and three things happen, in increasing order of how much they ruin the output:

1. It sounds rushed. Not chipmunk-pitched — the pitch is preserved — but the cadence is wrong in a way listeners immediately read as artificial.
2. Consonants smear. Time-stretching handles steady vowels gracefully and plosives badly. Hindi's aspirated/unaspirated distinctions (क/ख, त/थ) live exactly in the burst timing that gets destroyed.
3. The error compounds across the segment, not within it. You can fit each chunk into its own window and still have every chunk sound wrong.

The deeper objection is architectural. Time-stretching is a repair applied to the last stage of the pipeline for a decision made in the second stage. Any time you find yourself repairing a downstream artifact of an upstream free choice, the real fix is to constrain the upstream choice.

Stage 2 — During speech generation (native duration control)

This is where the state of the art now sits. In February 2026, Sarvam AI shipped Sarvam Dub, and their write-up diagnoses the problem in almost exactly the terms I'd arrived at independently — a two-second English phrase might take three seconds in Hindi, or one and a half in Tamil. Their fix is to move the timing decision inside the model that produces the voice: you hand the model a target duration up front, and it generates speech that lands on that duration in the first place — nothing is stretched, because nothing needed fixing.

My own v2 moved here too. The rewritten audio_sync.py docstring records the shift:

NEW: IndicF5 generates audio at the correct duration natively.
     This module just places audio chunks at the right timestamps on a timeline.

The audio module stopped being an audio engineer and became a filing clerk: drop each clip at its timestamp, fade 50 ms at the seams so the joins don't click, mix the original music and background back in underneath.

Stage 3 — Before any audio exists (constrain the text)

Here's the gap I think stage 2 leaves open. A voice model with duration control still has to absorb whatever length the translator handed it. If the gap in the video is two seconds and the translated sentence is a three-second sentence, the model has to come in at two seconds anyway — by talking faster, by flattening the emphasis, by giving up the pauses that make speech sound considered rather than recited. The visible artifact is gone; the underlying conflict has just been moved somewhere you can no longer see it.

And that conflict is not small. I measured it. Base Llama-3.1-8B translating English into 11 Indic languages runs 29% long on average. That is the burden a stage-2 system quietly eats on every single segment. So: don't shorten the audio, and don't make the voice model do all the squeezing either. Ask for a shorter sentence.

These stages are complementary rather than competing — my pipeline runs stage 2 and stage 3. Constraining the text means the speech model is handed something that already roughly fits, so it's making a 5% adjustment instead of a 30% one, and 5% is an adjustment you can't hear.

02Constraining the translator instead

The unit of duration in speech isn't characters or words or tokens. It's roughly phonemes — the actual sounds. Two sentences with the same phoneme count take approximately the same time to say, across languages, far more reliably than two sentences with the same word count. That gives a target the translator can be held to. For each segment: take the English source and the reference translation, run the reference through a grapheme-to-phoneme converter (I use espeak-ng) to get its phoneme count N, and train the model on a prompt that includes N.

The prompt format is deliberately plain. This is a real row from val.jsonl, verbatim:

[Translate to Hindi] [Target Phonemes: 39] "Milk is rich in calcium."

There is no special loss function here, and that's the part I find most interesting. No auxiliary length head, no differentiable length penalty, no reinforcement learning against a duration reward. The phoneme budget is just text in the prompt, and the model learns to respect it implicitly through ordinary next-token cross-entropy — because across 53,000 training examples, the completions that follow [Target Phonemes: 39] are consistently 39-phoneme completions. You are teaching the model a correlation it can only exploit by actually modelling length. That's it.

The training setup

Everything here ran on Kaggle's free tier: two Tesla T4 cards, which I did not pay for and cannot keep for longer than a few hours at a time. That single fact determines every choice below. An 8-billion-parameter model won't train on a T4 the normal way, so I stored the model at reduced precision to make it smaller, froze it, and trained a small set of extra weights alongside it — a few million instead of eight billion. That combination has a name, QLoRA, and the parameters below are the dials on it.

Eleven languages in one adapter, spanning two families that behave very differently: Indo-Aryan (Hindi, Bengali, Marathi…) and Dravidian (Tamil, Telugu, Kannada, Malayalam). Dravidian languages are agglutinative — they build long single words where English uses several short ones — so the relationship between tokens emitted and sounds produced is different for them. As I'll come back to, that means they don't learn this task on the same schedule.

03What it bought

Here's the headline table, comparing the un-fine-tuned base model against four checkpoints along the training trajectory. Four metrics, and it matters that they disagree.

Source: files_v3/evaluation/results/eval_out_all__eval_report.md

Length error — |N_generated − N_requested| / N_requested. The base model misses the requested phoneme count by 49.5% on average: it's effectively ignoring the instruction and translating naturally. The fine-tune brings that to 10.3%, a 4.8× reduction. A 10% phoneme error on a 1.5-second line is about 150 ms of drift — comfortably inside what crossfading and natural pauses absorb, which is the entire point.

Signed error tells you the direction of the miss, and this is the number that surprised me most. The base model sits at +0.290: it systematically over-generates. That single number explains why every dubbing pipeline needs a compressor of some kind — the translator's untrained bias is always toward too long. After fine-tuning the bias is roughly −0.04: slightly short. Slightly short is the far friendlier failure mode, because you fill it with silence rather than destroying consonants.

chrF++ is the one I expected to be the price of all this — the obvious worry is that a model told to hit a length budget hits it by throwing meaning away. It went up, 22.0 → 30.1. Good evidence that the length constraint isn't being paid for out of meaning — but not proof, and the gap between those two things turned out to matter.

Length slope is the cleanest probe of the four. Ask for one sentence at 0.6, 0.8, 1.0, 1.2 and 1.4× its natural length and plot what you got against what you asked for. Perfect obedience is a slope of 1.0; ignoring the budget is a flat 0. The base model sits at 0.593; the fine-tune reaches 0.687 — real movement toward obedience, and the metric I'm least satisfied with. The model has learned to care about the budget without learning to hit it, and closing that gap is the open problem.

04But did it still mean the same thing?

There is an obvious way to hit a phoneme budget that I'd been quietly hoping the model didn't discover: delete something. Drop the subordinate clause and you'll land on any target you like. The sentence will be shorter, fit the window perfectly, and be wrong. chrF++ is real evidence against this, but weaker than it looks — it rewards using the same words as the reference, and crucially it requires a reference translation, which exists in evaluation and never in production. So the check has to happen twice, in two places, for two reasons.

At training time: don't let the model learn the shortcut

To teach length elasticity, I generate paraphrases of each reference at 72% and 128% targets and add them as training rows. The hazard is that the paraphraser takes the same shortcut. So every generated paraphrase passes two gates before it becomes a training row:

1. Meaning: embed the original and the paraphrase with a multilingual sentence embedder and reject anything below 0.80 cosine similarity. This is what distinguishes "rephrased more tersely" from "deleted a clause" — the deletion moves the sentence measurably away from where it started.
2. Length: the paraphrase must actually have moved phoneme count by ≥10% in the intended direction, or it isn't teaching anything.

At inference time: the same guarantee, missing

Building that training gate is what exposed the second problem, and it's the more interesting half. Once you've written the rule down explicitly, it becomes obvious it's scoped to the wrong place — it governs how training data gets built and says nothing about dub time, when the model generates text nobody will ever compare against a reference. I went looking for the counterpart gate on the inference path. There wasn't one.

At dub time the pipeline generates three candidate translations per segment — at 100%, 85% and 65% of the phoneme budget — then a trained duration predictor estimates each one's spoken length and the best is selected. If none fit, it tightens the budget by 25% and retries, up to three times; a tight segment can end up at ~27% of its original budget. Selection scored candidates on timing alone.

The actual flaw is quiet and worse than "it prefers the shortest text" (it doesn't — it scores closeness to the target). Two candidates of identical duration score identically, whether or not one dropped a clause. Rephrasing tersely and deleting the subordinate clause produce the same phoneme count, so the same score. I built a test with two candidates of deliberately identical length, one faithful and one gutted. Under the old scoring they came out at +0.991 and +0.991 — an exact tie. The pipeline picked the good one only because it happened to be generated first.

So I added the same gate at selection. Each candidate is now scored against an anchor for meaning as well as against the clock, blended so the winner is the most faithful candidate that fits. With the gate, that same tied pair separates to +0.951 and −0.229. Three decisions in it weren't obvious: what to compare against (the pipeline's own full-budget candidate — the least-compressed translation, already generated, free); why at selection rather than during generation (the anchor doesn't exist until generation finishes); and what to do when everything fails (not crash — ship the closest candidate and flag the segment as degraded, with the score and budget in the log).

05The finding that changed how I trained

Around step 3200, the training loss stopped improving:

Flat. Global minimum at 3200, then noise. On a free GPU quota with 2,900 steps still on the plan, the textbook move is to early-stop and take checkpoint-3200. I almost did. Then I noticed that the number I was about to early-stop on is not the thing I actually want.

That loss is cross-entropy — overwhelmingly a question of which words, in which order. Whether the sentence came out to 47 phonemes or 61 is a tiny fraction of that signal. So the loss can go quiet while length control is still improving underneath it. Between steps 3200 and 3801: CE went 0.5056 → 0.5075 (flat, marginally worse), but length slope went 0.656 → 0.687 (still climbing). The slope can't be a restatement of the loss — it's measured by making the model actually generate and counting the sounds, whereas the loss is measured by showing the model the correct answer. Early-stopping at 3200 would have thrown away 600 steps of improvement on the thing I cared about, while congratulating me for being disciplined about a stand-in.

There's a corollary I'm still living with: the best-loss snapshot (3200) and the best-length-control snapshot (3801) are not the same snapshot. Rather than gamble on either, you can take the adapter weights from several checkpoints around the plateau and average them together — a model soup. It costs nothing at inference (you end up with one adapter again) and usually generalises better than any single snapshot, because every checkpoint here is the same adapter at different moments in one continuous run. I'm averaging steps 2800–3400.

06What broke

A silent resume bug that would have run forever. Because training was sliced across sessions, each new session had to restore the previous session's checkpoints from a mounted path. My glob was one directory level too shallow. It matched nothing, found no checkpoint, and cheerfully started training from step 0 — with a completely healthy-looking loss curve, because a fresh run's loss curve looks fine. Every session would have restarted forever. I caught it 10 minutes in by chance. Provenance logging catches this class of bug and loss-curve inspection categorically cannot.

Trusting the aggregate. Eleven languages across two families do not plateau simultaneously — Tamil's length control was still tightening while Hindi's had flattened. Every aggregate number in the table above is hiding that. I'd built the per-language decomposition into the eval script from the start and then read the aggregate column anyway.

Nearly scored with the wrong ruler. The whole approach rests on counting sounds, and the tool that converts spelling into sounds makes judgement calls. I used one tool to write the training labels and nearly used a second to score the results, which would have added a constant offset to every measurement — large enough to bury the effect. The habit to take from this: the function that creates your labels and the function that scores your outputs must be the same function, imported from the same module.

07What I'd do differently

Train the slope directly. Right now the model learns length control as a side effect. 0.687 is where a side effect gets you; the direct version rewards the model explicitly for both meaning and length and lets it optimise the trade-off itself.

Build the length-response probe on day one. It's the only metric here that measures the actual capability, it's cheap, and I built it last. Everything I learned late, I'd have known in week one.

Log per-language from the first eval, and read it. Building the decomposition isn't the same as using it.

The pipeline this feeds has since grown a duration predictor that acts as a referee: the LLM proposes three candidates at 100/85/65% of the budget, and the predictor measures each in real milliseconds and picks the one that fits. A generator that proposes and a referee that measures beats a generator that guesses.

Nirmit K. Tripathii — AI/ML Engineer & Football Data Scientist

Three years shipping NLP, agentic AI and multimodal systems. I write up real projects — including the parts that failed. Available now for part-time, consulting and research collaboration (remote), and open to full-time roles from September 2026. If you're working on speech, multilingual NLP, or sports analytics, I'd like to hear about it.