跳到正文
DEV Community· Asael Shinder·· 2 天前AI 評分50

更多產品中的 AI,或會在用戶裝置上運行

More of the AI in Your Product May Run on the User's Own Device

AI 導讀

越來越多產品的 AI 功能可在用戶的手機或手提電腦上運行,而非把文字或圖片送往數據中心的大型模型。Apple 的 AI 功能使用約 3 billion parameters 的裝置端模型,較難處理的請求才會傳送至伺服器。本機運行可保障私隱、節省成本,並讓功能在沒有訊號時繼續運作;小型模型適合狹窄任務,但不擅長長篇推理或廣泛知識。

正文

Asael Shinder

For the past few years, adding AI to a product has usually meant sending the user's text or images to a large model in a data centre and waiting for the answer. That is starting to change. More and more of it can now run on the phone or laptop in front of the user.

The hardware is arriving quietly. Many new phones and laptops include a chip designed for machine learning, often called a neural processing unit. Microsoft's Copilot+ PCs, introduced in 2024, require one capable of at least forty trillion operations per second. Apple's own AI features use a model of around three billion parameters that runs on the device, sending only harder requests to its servers. Google runs a small model, Gemini Nano, on some Android phones, and has been building ways for web pages to use a model inside the Chrome browser.

Why does it matter? Three reasons. Privacy, because the data never leaves the device. Cost, because every request you do not send to a server is one you do not pay for. And speed and reliability, because the feature still works on a train with no signal.

The small models are not as capable as the big ones. They are good at narrow tasks. Summarising a message, suggesting a reply, sorting photos, cleaning up a transcript. They struggle with long reasoning or broad knowledge. So the design question for many features will become which part can run locally and which part needs the server.

What should you learn now? Get comfortable with the idea that a model is a file you ship, with a size, a version and an update cycle, much like any other dependency. Learn the basics of making models smaller, such as quantisation, which stores the numbers inside a model with less precision. And think about the users who have older devices. A feature that works beautifully on a new flagship phone and fails silently on a four year old one is not finished.

Also, be honest in your product. If something runs on the device, saying so is a real advantage. If it does not, do not imply that it does.

The cloud is not going away. But the choice of where your AI runs is becoming a design decision you will be expected to make.

– Asael Shinder

來源:DEV Community · dev.to