跳到正文
DEV Community· Castov·· 14 天前AI 評分52

如何使用 WebGPU 和 Transformers.js 在瀏覽器本機執行大語言模型

Running Local LLMs in the Browser with WebGPU and Transformers.js

AI 導讀

Castov 示範如何使用 WebGPU 和 Transformers.js,在瀏覽器本機執行量化大語言模型,令提示詞無需經由 HTTP 傳送至伺服器。文中提到 Qwen 2.5、Phi-3.5 和 Llama 3.2 等 4-bit 或 8-bit 量化模型,並稱 1-billion parameter 的 4-bit 模型約需 700MB VRAM。

正文

Castov

The Evolution of Client-Side Neural Execution
For years, executing machine learning models required beefy cloud servers equipped with high-end enterprise GPUs. However, the stabilization of the WebGPU API across major browser engines (Chrome, Edge, Firefox) alongside optimizations in ONNX Runtime Web and Transformers.js has fundamentally transformed this landscape.

By exposing direct access to local GPU compute shaders without the overhead of WebGL hacks, WebGPU allows browsers to execute 4-bit and 8-bit quantized models like Qwen 2.5, Phi-3.5, and Llama 3.2 directly inside the client sandbox.

import { pipeline } from '@huggingface/transformers';

// Initialize local model pipeline with WebGPU acceleration
const generator = await pipeline('text-generation', 'onnx-community/Llama-3.2-1B-Instruct', {
device: 'webgpu',
dtype: 'q4',
});

const output = await generator('Explain zero-telemetry architecture:', {
max_new_tokens: 128,
temperature: 0.2,
});
console.log(output[0].generated_text);
Memory Footprint and Quantization Trade-offs
Running models on client hardware requires balancing context window depth against browser memory limits. A 1-billion parameter model quantized to 4 bits requires approximately 700MB of VRAM, making it fully viable on mid-range laptops and workstations.

Key considerations include handling shader compilation latency during initial model load and utilizing browser Cache Storage APIs so weights are downloaded only once.

Security and Zero-Data Exfiltration
The greatest benefit of local AI execution is data sovereignty. When processing internal company logs, proprietary formulas, or confidential documents, the prompt vectors never traverse an HTTP connection, rendering man-in-the-middle attacks and cloud provider retention policies completely irrelevant.

Running Local LLMs in the Browser with WebGPU and Transformers.js | Castov Engineering Blog

How to leverage browser-native WebGPU acceleration to run quantized open-weight language models client-side with zero server roundtrips.

favicon castov.com

來源:DEV Community · dev.to