如何用 0.99 Loss-Trajectory Test 檢驗新硬件上的訓練是否正確

TL;DR: When someone claims "we trained a model on hardware X," a loss curve that goes down is not evidence. The measurement that settles it is a controlled comparison: train the same model under identical initialization, data, and schedule on the new hardware and on a trusted GPU reference, then correlate the loss trajectories. A recent inference-only-NPU pretraining run (VKORE-46M-NPU) reports a 0.99 correlation. Here is why that is the right thing to ask for.

Disclosure: VKORE-46M-NPU is from VIDRAFT, a technology partner of QuantID.

The weak evidence everyone shows

"Loss went down on our new accelerator" is the most common and least informative claim in hardware-training announcements. It can be true and still hide a broken training path:

• Low-precision kernels can shrink or clip gradients, so the model learns a biased version of the objective while loss still trends down.
• A subtly wrong backward can optimize a different effective loss and still decrease the reported one.
• Early-step loss drops are dominated by easy structure (token frequencies), which almost any gradient signal captures.

A single falling curve under-determines correctness. You need a reference.

The test that actually discriminates

Train twice under matched conditions: same seed and initial weights, same data and ordering, same optimizer and schedule, one run on the new hardware and one on a known-good GPU. Then compare the two loss trajectories across steps. If the new hardware does numerically the same work, the curves track each other and the correlation approaches 1.0.

The VKORE-46M-NPU run reports a 0.99 loss-trajectory correlation between a 32-chip inference-only NPU and a GPU reference, with comparable throughput. That paired, controlled evidence is far stronger than an isolated curve.

What a correlation does and does not tell you

• It does tell you: the training step is numerically faithful to the reference.
• It does not tell you: that wall-clock speed is equal, or that it scales to much larger models. Report those separately rather than folding them into one headline.

The released model is 46M parameters on TinyStories, a deliberately small proof of the training path. Whether it holds at billions of parameters is an open, measurable question.

Takeaway

Evaluating a hardware-training claim? Ask for the paired comparison and the trajectory correlation, not the lone loss curve. It is cheap to run, hard to fake, and it separates "the loss went down" from "training is correct."

Model and comparison data: https://huggingface.co/FINAL-Bench/VKORE-46M-NPU