Superhuman Mail 如何在 100 毫秒互動預算內設計 AI 功能

Superhuman's full drafting call will not finish inside a 100 millisecond interaction budget. That budget is the immediate UI response, and completed generation is separate. Prefetch context that does not depend on this turn, such as how this person writes to this recipient and what the thread already says, before they speak. Send only the new transcription when they stop. Prove the work on the most capable model you can call. Once people keep using it and the job is narrow, move that task to a smaller or fine-tuned model. Leave send on a human click.

Loïc Houssier, CTO of Superhuman Mail, described that constraint on Chain of Thought in May 2026. He had joined Superhuman a year and a half earlier. At 100 milliseconds the product cannot depend on the network for every step, so it works offline by default, with logic on the desktop and the phone and async jobs that sync when the device is online. Email fits that pattern because it is asynchronous. He was plain that inference under 100 milliseconds does not work, so the team changed the shape of the call.

How do you keep an AI feature inside a 100ms feel?

You pull prepared context out of the waiting path and send a short payload when the user finishes.

Superhuman had just released voice on mobile. The team was not satisfied with the speed. If transcription drags and a model then rewrites for voice and tone, people will not use it, because it feels unlike the rest of the product.

Speech to text was the easier part. On-device transcription is one way to do it. The hard part is a draft that frames intent after people correct themselves, and that matches the thread and the tone used with that person. Answering your mom and answering your boss are different.

Much of that context can sit ready before the dictation. They cached so the live step was a transcription payload, not one call with a huge context. That caching has a cost: open voice and then type anyway, and the context was sent for nothing.

A minimal sketch to adapt, not a drop-in library.

CONTEXT_CACHE = {}

# proving: capable model. specialized: one task, after the feature sticks.
FEATURE_ROUTE = {
    "voice_draft": "proving",
    "auto_label": "specialized",
}

def model_for(feature):
    stage = FEATURE_ROUTE[feature]
    return frontier_model() if stage == "proving" else specialized_model(feature)

def warm_context(user_id, thread_id, feature):
    context = {
        "tone": load_voice_and_tone(user_id, thread_id),
        "thread": load_thread_summary(thread_id),
    }
    # Preload on the model. Keep only a session reference on the client.
    session_id = call_model(model_for(feature), context=context, cache_only=True)
    CONTEXT_CACHE[(user_id, thread_id, feature)] = session_id
    return session_id

def run_feature(feature, user_id, thread_id, payload):
    session_id = CONTEXT_CACHE[(user_id, thread_id, feature)]
    # Live call is the transcription payload plus that session, not the full context.
    return call_model(model_for(feature), session_id=session_id, payload=payload)

So instead of having one call with a huge context, we pre- I would say load the context as much as we can and boom, then we send just like a payload of the transcription.

Loïc Houssier, CTO at Superhuman Mail, on Chain of Thought ep 59

How Superhuman Built AI Into a… - Chain of Thought | AI Agents, Infrastructure & Engineering - Apple Podcasts

Podcast Episode · Chain of Thought | AI Agents, Infrastructure & Engineering · May 22 · 50m

podcasts.apple.com

When should a feature leave the frontier model?

When people still want the feature, and when you can name the narrow job a smaller model will own.

They start with one expensive model, the best they have, to see if the feature can work and how it feels. Then each use case gets its own model. The agentic framework runs on the latest version of Opus. The sequence is to try the expensive model, make it useful, then make it efficient.

Auto-label shows the handoff. The job is classification: FYI, pitch, partner, newsletter, and the rest. Auto-label started on an expensive frontier model, run on every email, which was expensive. Once they knew which 10 or 12 labels would be useful, he said, you can fine-tune dedicated BERT models and maybe run inference on dedicated infrastructure built for cheap inference at scale. With the labels known, he described it as a typical BERT classifier job: an open model, fine-tuned a bit, on cheaper dedicated infrastructure rather than a general LLM provider, and good enough for those use cases.

Why did repeating the prompt at both ends stop helping?

That pattern was a fix for an earlier model generation, and it helps less on newer ones.

On those earlier models, repeating the request at the start and at the end worked well. The assumption was that those models weighted the beginning and the end of the message more than the middle, so copying the request into both ends lifted the middle, and they saw better results. On newer models, repeating the message does not help as much. Where you place the instruction still matters. Sometimes it goes at the end.

Repeating the message was surprisingly good for 4 models. It doesn't work as much on newer models.

Loïc Houssier, CTO at Superhuman Mail, on Chain of Thought ep 59

Why auto-draft the email and leave send to the user?

Send carries more risk than the draft, so the draft waits for the person.

They do not send email in the user's name, because it is too risky. The product auto-drafts, and the person hits send. Users were not ready to let that go.

If the agent can send mail or move money, the safeguards around it are probabilistic, and at some point it will mess up. His rule: if you want a real gate, you don't expose that action to the agent. He already had scars from giving an agent too much room in his own notes.

We do not send, we auto-draft. It's here for you to just hit send, but we won't send any mail for you, for now.

Loïc Houssier, CTO at Superhuman Mail, on Chain of Thought ep 59

FAQ

Can you make the model itself return in under 100 milliseconds?

Loïc said that does not work. Prefetch voice, tone, and thread context so the live request is mostly the new transcription.

When is a fine-tuned classifier enough?

After a capable model has shown the task is useful and the label set is known. Loïc described auto-label that way: an expensive model on every email first, then, once they knew which 10 or 12 labels would be useful, a fine-tuned open BERT-style classifier on cheaper dedicated infrastructure. The agentic framework stayed on the latest Opus.

Should the agent send once drafts look reliable?

Draft quality and permission to send are separate. This team auto-drafts and does not send on the user's behalf for now. Any safeguard around an agent that can send is probabilistic. Loïc's real gate is not exposing send to the agent at all, so leave send with the person.

Takeaway

• Write the interaction budget down (here, 100 milliseconds) before you design the model call.
• Prefetch recipient tone and thread context before the user speaks. Send the transcription as the live payload.
• Start on the most capable model. Move to a smaller or fine-tuned model only after the task and the labels are stable.
• Re-measure prompt tricks after a model change. A pattern that helped on an earlier model may not help on the next one.
• Auto-draft. Keep send, and any other hard-to-undo action, on a human confirmation.

The full conversation, with the transcript, is on Chain of Thought.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.