跳到正文
DEV Community· devs·· 13 小時前AI 評分67

Thinking Machines Lab 發佈 Inkling 2 Mini,Windows 即時智能體仍受音訊擷取路徑延遲限制

Inkling 2 Mini Lands as Laptop-Sized Looped MoE — Local Agentic Reasoning Still Waits on the Capture Path

AI 導讀

Thinking Machines Lab 於 2026 年 11 月 20 日發佈開放權重模型 Inkling 2 Mini,總參數量為 26.4B、啟用參數量為 3.4B。

正文

On 20 November 2026 Thinking Machines Lab released Inkling 2 Mini. It is an open-weights looped transformer (Apache 2.0) with 26.4B total parameters and 3.4B active. The middle twelve of its twenty-four layers run three times per token, so each token effectively sees ~5.4B weight activations and forty-eight layer visits. The checkpoint sits at roughly 20 GB in native NVFP4/MXFP8, small enough for a 32 GB Mac or a single desktop GPU. It leads open-weights models under 100B total parameters on DeepSWE v1.1 (48.2 %) and posts strong numbers on SWE-bench Verified (bash-only 79.2 %).

That is the kind of model you can drop next to a local coding agent or a short suggestion loop without renting a cluster. For developers who already ship real-time tools on Windows—overlays that surface translated suggestions during a technical interview or a sales call—the interesting question is not “can it reason,” it is “how fast does the first usable token arrive after the other party stops talking.”

Waveforms

What the loop actually buys you

A normal dense 27B model spends the full parameter budget on every token. Inkling 2 Mini reuses the same middle blocks, so the memory traffic per token stays closer to a 5–6B model while the computational depth matches a deeper network. Thinking Machines reports it runs roughly four times faster and six times cheaper per task than Qwen3.8-27B on their agentic coding suite. GGUF builds via llama.cpp and Unsloth already exist; Ollama and EXO list it for Apple silicon.

Practical numbers for a local agent:

  • Context up to 1 M tokens.
  • Multimodal (text + image).
  • Function calling / tool use is listed as supported.
  • First-token latency on a 32 GB Mac or a mid-range NVIDIA card is still in the hundreds of milliseconds once you add KV cache and the three loop passes.

That is fine for an offline coding agent that can think for a few seconds. It is not free for a live overlay that needs a polished suggestion before the next pause in conversation.

The capture path is still the dominant term

None of the model improvements touch the Windows audio stack. A usable bilingual overlay needs two clean PCM streams that stay separate:

  1. Local microphone (what you say).
  2. Loopback of the render endpoint (what the other party says through Teams, Zoom, Meet, or the browser).

Both must remain in shared mode so the call client keeps the device. Exclusive mode breaks the call. The audio engine (audiodg.exe + audioeng.dll) mixes, applies communications enhancements, spatial audio, and vendor APOs. Typical shared-mode period is 10 ms. IAudioClient3 can request a smaller period; the engine frequently ignores it or falls back under load. Bluetooth (even LE Audio) adds its own buffering.

Measured end-to-end stack on typical hardware:

  • WASAPI shared-mode client (10–20 ms buffer) → engine mixing + enhancements → capture callback (copy into lock-free SPSC ring, return immediately) → framing + lightweight VAD + resample to 16 kHz mono → STT (local or remote) → small local model for translation polish or short suggestion → overlay paint on a dedicated thread (<16 ms).

Even when the suggestion model is local and fast, the first usable partial still lands in the 80–200 ms range after the other party stops. The 10–40+ ms the engine can insert before any sample reaches your process does not disappear because you swapped a 27B dense model for a looped 26B MoE.

Where a laptop-sized agent actually helps

The model is useful once the audio is already in your process. A practical architecture looks like this:

WASAPI dual-track (mic + loopback)
  → lock-free rings (SPSC, no locks in callback)
  → 20–40 ms frames + energy VAD
  → local STT (faster-whisper / Windows ML Task / small GGUF)
  → token stream or short text buffer
  → Inkling 2 Mini (or any 3–8B class model) for:
       - short translation polish
       - one-sentence suggestion
       - tool call if you need a local RAG lookup
  → overlay thread

You keep the heavy reasoning off the capture thread. The model can run on a second CUDA stream or on the NPU/GPU while the capture callback stays under a few microseconds. Tradeoff: the three loop passes increase memory bandwidth on the active layers. On a bandwidth-bound card that can show up as lower tokens/sec than a pure 4B dense model, even if the quality is higher.

Code sketch for the hand-off (pseudo, not production):

// capture callback — do almost nothing
void OnCapture(const BYTE* data, UINT32 frames) {
    ring.push(data, frames);  // lock-free
}

// worker thread
void Process() {
    auto frame = ring.pop_frame(20);  // ms
    if (!vad.speech(frame)) return;
    auto text = stt.partial(frame);
    if (text.empty()) return;
    // fire and forget to local agent
    agent.enqueue_suggestion(text);  // Inkling or smaller
}

Tradeoffs that actually matter

Choice Latency win Cost When it breaks
Cloud STT + local polish network RTT on STT API bill offline, privacy
Full local STT + Inkling polish no network on STT GPU VRAM ~20 GB + STT VRAM pressure, thermal
Tiny local model only lowest tail almost none quality on complex technical terms
Hybrid (local partial + cloud refine) first partial fast both complexity

For interview/call overlays the hybrid or full-local path with a small polish model is usually the practical one. Inkling 2 Mini is large enough that you may still prefer a 1–4B specialist for the suggestion step and keep the bigger model for offline prep or longer agent tasks.

What to measure next

If you are already shipping a dual-track overlay:

  • Pin the capture thread and drop enhancements. That reclaims the largest slice of the floor.
  • Measure partial-to-overlay latency with and without the new model. The loop passes should be visible in the tail.
  • Keep the suggestion prompt short and structured so the model does not waste tokens on preamble.
  • Verify the GGUF or native checkpoint actually fits with your STT model resident at the same time.

The model is a real step for local agentic work. It does not move the audio engine. The overlay that feels instant is still the one that owns the capture path and treats the model as a fast second stage, not the first.

I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.

來源:DEV Community · dev.to