For about a day, one of the three inboxes was invisible. The agent would answer "what's urgent across all my mail?" cheerfully and completely — from two accounts. The third, a Yahoo mailbox reached over IMAP, had quietly dropped out.
No error surfaced to the user. The logs, if you went looking, showed the same line on repeat: Unexpected close. The connection to that mailbox had died hours ago, and every query since had been talking to a corpse and calling it a failed account.
This is the unglamorous half of building an agent that consolidates other services: the consolidation is a weekend; keeping it alive is the product. A connection to a mailbox isn't an HTTP request you fire and forget — in this design it's a long-lived MCP session to a child process, and child processes exit. IMAP servers hang up idle connections. Laptops sleep. A provider server crashes and restarts. This article is the machinery that treats all of that as normal weather instead of an outage. (The hub design it builds on is Part 1.)
01Why "connect once and reuse" is a trap
The natural first design is the one everybody writes: connect to each provider at startup, stash the client, reuse it. It's efficient — no per-call handshake — and it works flawlessly in every demo, because demos run for ninety seconds on a good network. It has two independent failure modes, and this project hit both.
A boolean set once is a lie. The first version marked an adapter connected = true after the handshake and never revisited it. But "connected" isn't a fact you learn once; it's a state that changes without asking you. When the Yahoo child hung up, nothing flipped that boolean back. The adapter kept reporting itself healthy, kept routing calls into a dead pipe, and kept throwing Unexpected close — forever, because nothing in the code path could ever decide to reconnect.
One failure taking down the whole answer. The other naive instinct is to fan out with Promise.all. But Promise.all rejects the instant any promise rejects. So the moment one account has an expired token, the entire cross-account query throws, and a user with three healthy inboxes and one flaky one gets nothing from any of them. The blast radius of one bad account is all accounts.
Both bugs share a root: the code assumed a connection is a thing you have, when it's really a thing you maintain.
02Isolate, self-heal, and give up fast
Three mechanisms, each targeting one way the naive version failed.
1. Isolation, via allSettled. The fan-out uses Promise.allSettled, which waits for every account to either succeed or fail and hands back the outcomes side by side. Fulfilled accounts contribute their emails; rejected ones get logged and skipped. Three good inboxes and one dead one returns three inboxes' worth of mail, not an exception.
2. Self-healing, via drop-detection + reconnect-on-use. Instead of trusting a boolean, the adapter wires the MCP client's onclose and onerror events back to its own state: a drop flips connected = false, so the next call sees a disconnected adapter and transparently reconnects before running. And if a connection dies mid-call, the operation catches the specific "connection dropped" error, reconnects once, and retries. Reads are idempotent, so a single retry is safe.
3. Giving up fast, via a cooldown. Self-healing has a failure mode of its own: if an account is genuinely dead — wrong password, server down — retrying it on every query would stall the whole fan-out, because even a failed connect costs a timeout. So a failed connect starts a 60-second cooldown: during it, that account fails instantly. A dead account degrades to "temporarily absent," never "makes everything slow."
Two smaller pieces hold it together. A connection generation counter invalidates stale event handlers — when you tear down socket #3 and open socket #4, the old socket's late onclose must not flip the new one to disconnected. And an in-flight connect promise dedupes concurrent callers, so four tools hitting a reconnecting account share one handshake instead of racing to open four.
03The implementation
ensureConnected is the heart of it. Read it as three guards before the actual connect: already connected (do nothing), already connecting (join that attempt), recently failed (fail fast).
async ensureConnected(): Promise<void> {
if (this.isConnected()) return;
if (this.connectPromise) return this.connectPromise; // dedupe concurrent callers
const err = this.lastConnectError; // fail fast during the cooldown window
if (err && Date.now() - err.at < RECONNECT_COOLDOWN_MS) {
throw new ProviderConnectionError(this.provider, `Not reconnecting yet (cooling down)`);
}
this.connectPromise = (async () => {
try { await this.establishWithRetry(); await this.afterConnect(); this.lastConnectError = null; }
catch (e) { this.lastConnectError = { at: Date.now(), message: String(e) }; throw e; }
})().finally(() => { this.connectPromise = null; });
return this.connectPromise;
}The drop-detector is what makes "the next call reconnects" true. On every successful connect, the handlers below get wired — and each captures the generation it was born under, so a handler from an old socket bails the moment it notices it's been superseded.
const generation = ++this.connectionGeneration;
const onDrop = (reason?: unknown): void => {
if (generation !== this.connectionGeneration) return; // superseded by a newer connection — ignore
this.connected = false;
this.client = null;
this.onConnectionLost();
};
client.onclose = () => onDrop();
client.onerror = (e: Error) => onDrop(e);Mid-flight recovery lives in withConnection, which wraps every downstream call. The subtle part is isConnectionDrop: a reconnect-and-retry is only safe for errors that mean the pipe died, not for a normal tool-level failure ("no such message"). Retrying the latter would just repeat a legitimate error.
private async withConnection<T>(fn: () => Promise<T>): Promise<T> {
await this.ensureConnected();
try { return await fn(); }
catch (error) {
if (!isConnectionDrop(error)) throw error; // a real tool error — don't retry
await this.teardown(); this.onConnectionLost(); await this.ensureConnected();
return await fn(); // reads are idempotent → one retry is safe
}
}isConnectionDrop is deliberately a broad regex, because the same "the child died" reality surfaces under a dozen different messages depending on where in the stack it's noticed — the MCP SDK's Connection closed, or the raw socket's ECONNRESET / EPIPE / socket hang up / write after end.
04The dials, and why each is set where it is
| Dial | Value | Why this |
|---|---|---|
CONNECT_TIMEOUT_MS | 60,000 | Gmail's server refreshes OAuth on startup — ~30s observed. A tighter cap kills a healthy account mid-handshake |
RECONNECT_COOLDOWN_MS | 60,000 | After a failed connect, one dead account must not re-stall every fan-out. Fail fast for a minute, then allow another try |
CONNECT_RETRIES | 3 | Transient blips retry (0.5s, then 1s); auth failures fail fast — retrying a wrong password never helps |
| Urgent-notify dedup set | bounded 5,000 | Remember which urgent emails we've alerted on, without unbounded memory growth |
| Test suite | 49 tests, all green | Two exist only to reproduce the two production drops below |
The auth-failure carve-out matters more than it looks. Transient errors and permanent ones want opposite treatment — retry the timeout, fail fast on the 401 — so the connect loop throws immediately on anything matching unauthorized/forbidden/401/403/invalid token. Retrying a bad credential just burns the cooldown budget on an outcome that will never change.
05What broke
Both of the bugs this machinery exists for are pinned in a regression test, because a resilience feature you can't reproduce on demand is one you'll break next month.
1. Yahoo "Unexpected close" — and the twist that made it hard. The IMAP child's connection drops mid-request; the adapter must reconnect. Fine. But the IMAP MCP server holds its mail account in per-connection state — each fresh child process starts with an empty account store. So reconnecting isn't enough: the adapter has to re-provision the account into the new child before any read will work, or it reconnects successfully and then returns zero emails from a mailbox that's full. The fix is a per-provider afterConnect hook, and the test asserts exactly this:
adapter.dropNextGetLatest(); // next read kills the child mid-request
const recovered = await adapter.listEmails();
expect(recovered).toHaveLength(1); // read still succeeds
expect(adapter.connectCount).toBe(2); // reconnected to a fresh child
expect(adapter.addCount).toBe(2); // AND re-provisioned the account into it2. The connection that stayed "connected" forever. The boolean that never flipped back. The test drops the child between calls and asserts the adapter both notices and transparently recovers on the following call.
read. So account_status proudly reported zero unread on an inbox with forty. No crash, no log — a number that was confidently wrong. The failures that hurt in a multi-service agent are the silent ones, and the only defence is a test that asserts the value, not just the absence of an exception.06What I'd do differently
Surface degraded accounts to the user, not just the log. Today a cooling-down account is invisible in the answer — the digest is silently built from two inboxes instead of three, and only stderr knows. inbox_summary should carry a line like "Yahoo unavailable (reconnecting)." Isolation without disclosure is just a nicer way to hide a problem — the exact trap the Gmail-unread bug fell into.
Add a real health check. Connection liveness is currently proven only by attempting a real operation. A cheap periodic listTools ping would detect a dead child before a user query does, so recovery happens in the background instead of on the critical path.
The through-line for the whole series is here: the moment your agent orchestrates other services, their failure modes become yours. Isolation, drop-detection, and a cooldown aren't gold-plating — they're the difference between a thing that demos and a thing you can leave running while you get on with your day.
The transferable lesson
- A connection is a state you maintain, not a boolean you set once — wire drop events back to that state.
- Isolate accounts with
allSettled; reconnect on next use; cool down a failed connect so a dead account degrades to "absent," never "slow." - Only retry errors that mean the pipe died; fail fast on auth errors and real tool errors.
- In a multi-service agent the dangerous failures are silent — assert the value, not just the absence of an exception.
