Cider 量化：裝置端推理的工程取捨

Most conversations about on-device models start and end with one question: how much did quantization speed things up? That is the easiest question to answer and the least important. The trade-offs that decide whether a quantized agent is usable sit where no speed benchmark reaches. Here are the ones we hit building Cider, our on-device inference layer.

The speedup is smaller than the compression ratio suggests
Weight quantization feels dramatic. Take a model from FP16 to INT4 and the weights occupy roughly a quarter of the memory. That part is real.

Latency is different. A desktop agent runs a pipeline: capture the screen, encode the visual input, assemble context, decode the next action, parse it, execute it. The forward pass is one segment of that pipeline, and weight reads are one part of the forward pass.

On many edge chips, memory bandwidth and kernel scheduling are the harder constraints. If decoding is autoregressive and dominates wall-clock time, that cost does not shrink with model size. A model 4x smaller is rarely 4x faster, and sometimes barely faster at all.

When someone quotes a speedup, the useful follow-up is: prefill or decode, single turn or full episode? Those two framings differ by an order of magnitude.

The real risk is that behavior boundaries move
Speed is measurable. Boundaries are not. You find out by watching something fail.

Run the same GUI tasks on FP16 and INT4. On click, type, scroll, the two are indistinguishable; any gap sits in the noise. Then look at specific categories and the split appears:

Dense small-text interfaces. Quantization error amplifies in the visual features, and a checked box gets read as unchecked.
Toolbars with similar, tightly packed icons. Low precision conflates neighbors more often.
Multi-step decisions over long context. Each step is slightly off, and errors compound. Divergence at step five becomes a runaway by step ten.
These are the daily-use cases. They are underrepresented in standard benchmarks, so an aggregate score drops a point or two and still looks fine. In real use they are what people keep hitting.

Activations are the harder half
Weight quantization is comparatively mature: stable distributions, developed toolchains, static and analyzable error. Activation quantization is where it gets difficult.

Activations depend on input, so the distribution is dynamic, wide, and carries outliers. For a visual model, the spread across resolutions and interface densities is far larger than for text. Coarse activation quantization does not spread error evenly across tokens; it concentrates on the harder inputs. The situations needing judgment most are exactly where precision degrades most.

Our choice: keep activations more conservative than weights. Weights get pushed harder, activations keep headroom. The model ends up slightly larger than it could be, but the boundaries hold. There is no universal answer; it depends on how much your task fears edge cases.

The calibration set sets the ceiling
Algorithm choice matters less than most comparisons suggest. Calibration data matters more.

Calibration is what the quantizer uses to estimate activation distributions and fix parameters. Calibrate on generic text or natural images and the parameters lean toward those distributions. But an on-device agent sees desktop screenshots: windows, toolbars, input method boxes. Calibrate against the wrong distribution and the parameters are off from the start; no tuning recovers it.

We sample calibration data from real task traces: different resolutions, different layouts, light and dark themes. Small volume, but it covers the distribution we actually meet.

Evaluation has to change too
A generic text benchmark is driving with your eyes closed. It cannot measure accumulation across an episode, or precision loss on visual input.

Three layers: per-step accuracy, which localizes where precision was lost; multi-step completion rate, which captures accumulation (95 percent per step becomes roughly 60 percent over ten); and recall on critical interface elements, a small share of any aggregate score and the most visible to users.

A counterintuitive finding: the same quantization config varies more across tasks than different configs vary on the same tasks. Understanding your task distribution matters more than tuning parameters.

Closing
Quantization is not about shrinking numbers. The speedup is measurable, the boundary shift is not; weights are easy, activations are not; the algorithm is easy to pick, the calibration set is not. What makes a scheme production-ready is whether it holds the line on the tasks you fear most.

The full on-device agent pipeline is open source: https://github.com/Mininglamp-AI/Mano-P