Foundry Local 與 Qwen3.8-27B 讓 GitHub Copilot 可本機運行，Windows 音訊仍主導即時延遲

Microsoft’s Foundry Local blog on 5 November 2026 put a concrete local agent path in front of every developer with a 24 GB+ NVIDIA GPU. Load the 4-bit Qwen3.8-27B (~18 GB), point GitHub Copilot’s Bring-Your-Own-Key provider at the local OpenAI-compatible endpoint, and the model picker shows a private, offline, zero-per-token coding assistant. The release notes and CLI steps are short. The practical surface is larger.

Qwen3.8-27B is a dense 27 B vision-language model with a 262 K native context, thinking mode, and tool-calling. Microsoft optimized the full pipeline on ONNX Runtime + CUDA: DFlash2 speculative decoding, CUDA graph fusions tuned for SM121, INT4 weights, KV-cache quantization, prefix caching, and continuous batching. On an RTX 5090 they measured 1.5–1.9× generation speed versus llama.cpp across prompt lengths from 4 K to 260 K tokens, with the gap widening past 131 K — the regime of repository-scale agent work. Terminal Bench 2.1 sits at 73.0, SWE-bench Pro at 61.7. For everyday coding loops this is now a real option rather than a demo.

The local endpoint in practice

The CLI path is three commands after the WinGet install:

winget install Microsoft.FoundryLocal
foundry model load qwen3.8-27b
foundry server status

foundry server status prints the base URL (usually http://127.0.0.1:<port>). In the Copilot app you add a Foundry Local provider, paste the URL, pick Qwen3.8-27B from the catalog, and switch the model. CUDA and cuDNN must be on PATH; Foundry Local does not bundle them. Once the server is up the OpenAI-compatible /v1/chat/completions and tool-calling endpoints are live. Prefix caching means repeated repository context does not re-prefill from zero. Continuous batching lets the same process handle multiple agent turns without serializing everything behind a single lock.

On Windows the binary already carries the usual application manifest. That removes one class of “why won’t the CUDA DLLs load under a scheduled task or overlay process” failures. The remaining work is the usual: pin the DLL search path, keep the service account’s PATH correct, and make sure the GPU is not already saturated by a game or another inference server.

The numbers that do not move

None of the above changes the dominant term once the workload leaves pure text generation and enters real-time audio. A local 27 B model that can emit a coding patch in a few hundred milliseconds is still useless for a live interview or call if the microphone path has already burned 80–120 ms of buffering and the loopback path has added another 40–60 ms before the first sample reaches your process.

Typical measurements on a recent Windows 11 machine with an onboard Realtek and a USB headset still show:

• Exclusive-mode WASAPI capture: 10–20 ms buffer + driver
• Shared-mode / loopback: 30–80 ms depending on the audio engine mix period
• Bluetooth or USB class driver: another 20–50 ms
• Process isolation around an overlay: extra context switches and the kernel’s audio engine queue

The model is no longer the long pole; the OS audio graph is. Speculative decoding and INT4 help the GPU side. They do nothing for the WASAPI shared-mode period, the Bluetooth SCO/A2DP latency, or the fact that a global hotkey overlay must still pull samples through the same engine.

Architecture that actually ships

A practical local speech pipeline on Windows therefore looks like this:

1. Exclusive or low-latency WASAPI capture (or a carefully tuned loopback) into a small ring buffer.
2. VAD or simple energy gate so you do not feed silence into the model.
3. Streaming STT (whisper.cpp, Nemotron Transcribe, or a small local model) that can emit partials.
4. Optional local LLM post-process (the same Foundry Local endpoint or a smaller model) for cleanup, translation, or prompt formatting.
5. Overlay that injects the result at the cursor or into the call without extra buffering.

The capture and injection steps dominate. Everything else can be swapped. That is why tools that own the Windows audio path and the overlay — rather than just wrapping another OpenAI-compatible endpoint — still matter once you leave pure coding agents.

Tradeoffs that remain

• 24 GB VRAM is the floor for the 4-bit 27 B model with a usable context. Drop to a smaller Qwen or a different runtime and you lose the agentic scores Microsoft published.
• Foundry Local’s telemetry and model catalog are Microsoft-controlled. The OpenAI-compatible surface is real; the model download and optimization recipe are not fully open.
• Hybrid routing (Copilot Fusion choosing local vs cloud) is still experimental. For latency-sensitive work you still want the local endpoint pinned.
• Real-time audio remains a separate problem. No amount of speculative decoding fixes a 100 ms capture buffer.

Local coding agents just crossed a practical threshold. Local real-time speech tools have not. The capture stack, the overlay, and the end-to-end latency budget are still the parts that require hands-on Windows work rather than another model download.

I'm building Talkaside, a Windows app for job interviews and calls in another language.
Drafted with AI assistance from this project's code and logs.