The Model Spent Its Whole Budget Thinking

The categoriser returned nothing. Not an error, not bad JSON — an empty string.

The task was trivial. Hand the model an email, get back a small JSON object: a category, an urgency score, a boolean, a one-line reason. Eighty tokens of output, maybe. I'd set the output budget to 512, which felt almost wasteful — six times the size of the answer.

Every third or fourth email came back blank. The response's finishReason was MAX_TOKENS — the model had hit its output ceiling — on an object that would have fit six times over. The obvious reading is that the model wrote a giant blob and got cut off. It hadn't. It had written nothing at all and still hit the ceiling. The model was spending the entire budget before it wrote a single character of JSON. It was thinking. (This runs inside the enrichment layer from Part 1.)

01Why "size the budget to the output" is exactly backwards here

There's a rule of thumb everyone internalises: set max_tokens to roughly what you expect back, with some slack. It's good advice, it saves money, and on a classic model it's correct. The default model here was a thinking model — one of the current generation that does a hidden chain of reasoning before it emits the answer you asked for. And the thing nobody tells you until it bites: those hidden reasoning tokens are spent first, and they count against the same output budget as the visible answer.

So the accounting on my "wasteful" 512-token budget actually looked like this: ~350–460 tokens of invisible reasoning, spent first and billed to the output budget, leaving 50–160 tokens for the actual JSON. An 80-token object needs ~80. Sometimes it fit. Often it didn't, and the model hit MAX_TOKENS mid-thought, before the JSON started — so the visible output was empty.

Read that again, because it inverts the rule of thumb completely. On a thinking model, a too-small max_tokens doesn't truncate your answer. It deletes it. And it fails in the worst possible way: silently. An empty string isn't an exception. Unless you're checking finishReason, it looks exactly like "the model chose to say nothing."

02The approach: floor it, then double on retry

Two facts make the fix straightforward once you see them. First, the budget is a ceiling, not a target. If the model finishes its JSON and emits a stop token, generation ends there — you are not billed for the headroom, and you don't wait for it. A generous cap costs nothing on the calls that don't need it. Second, when a JSON call comes back empty or truncated, the correct response is a bigger budget, not a retry with the same one. Reasoning length varies per input, so the fix is to escalate: floor the budget well above the visible output, and if the reply still comes back starved, double it and try again.

03The implementation

The core is a small retry loop. The critical line is the definition of starved: an empty body or a hard token cut-off both mean "reasoning ate the budget," and both should trigger a bigger retry rather than a parse attempt on nothing.

let budget = Math.max(options.maxTokens ?? this.config.maxTokens ?? 0, JSON_TOKEN_FLOOR);

for (let attempt = 1; attempt <= maxAttempts; attempt++) {
  const response = await this.complete({ ...options, maxTokens: budget, jsonMode: true });
  const text = (response.content ?? '').trim();

  // Empty body OR a hard token cut-off means "reasoning ate the budget".
  const starved = text.length === 0 || response.finishReason === 'MAX_TOKENS';
  if (!starved) {
    const parsed = extractJson(text);
    if (parsed !== undefined) return validator ? validator(parsed) : (parsed as T);
  }
  if (attempt < maxAttempts) budget = Math.min(budget * 2, JSON_TOKEN_CEILING); // 2048→4096→8192
}
throw new LLMError(`LLM did not return valid JSON after ${maxAttempts} attempts …`);

JSON_TOKEN_FLOOR = 2048 and JSON_TOKEN_CEILING = 8192 are the two numbers that matter, and the comment above the categoriser's request records why 2048 and not, say, 600:

// THINKING model: ~350-460+ hidden reasoning tokens are spent BEFORE any JSON is
// emitted, and they count against this budget. 512 was too tight (thinking overran
// it → empty/MAX_TOKENS → failures). The cap adds no latency (generation stops at
// STOP), so keep generous headroom.
maxTokens: 2048,

The tolerant parser is the other half. Models wrap JSON in fences, prepend "Here is the object:", and leave trailing commas. extractJson strips the fence, walks the string to the first balanced {…} or […] (respecting quotes and escapes, so a brace inside a string doesn't fool it), and only then parses — with one light repair pass for trailing commas.

const sliced = sliceBalanced(s);           // first balanced {…}/[…], ignoring surrounding prose
if (sliced) s = sliced;
for (const candidate of [s, s.replace(/,\s*([}\]])/g, '$1')]) {  // as-is, then drop trailing commas
  try { return JSON.parse(candidate); } catch { /* try next */ }
}
return undefined;                          // unbalanced (truncated) → caller retries with a bigger budget

There's a nice property in that last line: if the object is truncated, sliceBalanced never finds a closing brace, returns undefined, and the caller escalates the budget. The same code path that handles "empty" handles "cut off halfway." One mechanism, both failure shapes.

04The dials, and why each is set where it is

Every value here was set by a failure, not a guess.

The concurrency row is the same lesson wearing a different hat. On the free tier, four enrichment calls at once produced zero rate-limit errors; eight produced them constantly. And when a 429 hands back a "retryDelay":"7s" hint, the client parses it out and waits exactly that long instead of guessing:

const m = msg.match(/"?retryDelay"?\s*[:=]\s*"?(\d+(?:\.\d+)?)s"?/i);
return m ? { retryable: true, retryAfterMs: Math.ceil(parseFloat(m[1]!) * 1000) } : { retryable: true };

05What broke

The token bug was bad; the way it hid was worse. The categoriser has a rule-based fallback — a keyword matcher — for when the LLM is genuinely unavailable. Graceful degradation, exactly as designed. Except the token starvation didn't look like "the model is down." It looked like a successful call that returned an empty string. So the categoriser caught the parse failure, shrugged, and fell through to keywords — for every email. The whole inbox quietly degraded from an LLM that understood context to a keyword matcher that didn't, and nothing anywhere reported an error. Everything that didn't hit a keyword became the default: informational.

If everything is informational, the LLM calls are probably failing (check stderr).— the tell that made it into the troubleshooting guide

The cost wasn't dramatic — no crash, no data loss. It was worse than dramatic: a system that looked like it was working, producing plausible-but-dumb categories, for as long as nobody looked closely. The lesson I actually took: a fallback that's too graceful will hide the failure it's falling back from. The fix was two lines — treat empty/MAX_TOKENS as retry-and-escalate rather than as a quiet failure, and log finishReason on every JSON call so "the model thought itself out of budget" is visible instead of inferred.

06What I'd do differently

Log finishReason and token usage from the first call, not after the incident. The bug was diagnosable in five seconds once those two fields were in the logs — finishReason: MAX_TOKENS with completionTokens near the cap and an empty body is an unambiguous signature. I flew blind for an evening because I was logging the content and not the metadata about the content.

Reach for a non-thinking model for pure structured extraction. Categorisation doesn't benefit from a hidden chain of thought; it just pays for it in latency and in exactly this failure mode. The thinking model earns its keep on explain_email and the digest narrative, where reasoning actually improves the output — so the real answer is to route by task, not to pick one model for everything.

The transferable rule is blunt enough to tape to a monitor: on a thinking model, size your token budget for the thinking, not for the answer — and never treat an empty completion as a choice the model made.

Nirmit K. Tripathii — AI/ML Engineer & Football Data Scientist

Three years building NLP, agentic AI and multimodal systems in production. I write up real projects — the architecture, the numbers, and the parts that failed. Available for part-time, consulting and research collaboration (remote).