重現研究顯示，預防模型崩潰所需的新數據率是恢復所需的 13.4–27.6 倍

If you train on your own output, you have probably read a number for "how much fresh data you need". The published range is roughly 1 % to 30 %. That spread is usually read as disagreement — one of those numbers must be wrong.

I spent a week reproducing a paper that argues something different: the spread is not disagreement, it is three questions wearing one coat, and the number everyone quotes is not even the quantity it claims to be.

Here are the four things I now check before I believe any fresh-data fraction, each with the measurement behind it.

1. The constant quoted as a rate is a lower bound on a different quantity

The folk closed form is (1 − p*)ⁿ: the probability a symbol of true frequency p* is missed in n draws. It gets quoted as "the per-generation loss".

The loss kernel is ℓ_i(q) = (1 − λp* − (1−λ)q_i)ⁿ, which is convex in q_i. The loop's stationary mean is exactly p* (the recursion E[q_{t+1}] = p* + (1−λ)(E[q_t] − p*) is exact), so Jensen gives

E[ℓ_i(q)] ≥ ℓ_i(E[q]) = (1 − p*_i)ⁿ     for every i and every λ > 0

So the folk form is a lower bound on the unconditional loss probability. But what a loop actually experiences is the conditional rate — the chance that a symbol now present disappears — and that is a different average, over a different set of states.

Measured against direct simulation, over 602 per-symbol readings that pass an adequacy gate (density × rate ≥ 30, so each rests on at least 30 expected loss events):

Median bound-to-measured ratio 0.08437 — the measured rate is a median 11.9× above the bound. Minimum 2.83e-47. And in the 83 cases it goes the other way, by up to 6.03×.

Which side a reading lands on is not noise. It is predicted by how often the symbol is absent:

For a symbol your loop keeps almost always, the folk constant understates the loss by more than an order of magnitude. For a symbol that is usually gone, it overstates it. It errs in both directions, so it is not a conservative bound you can tighten — it is the answer to a different question.

The disagreement also grows with n, because the closed form collapses exponentially while the measured loss does not: at λ = 0.01 on the worst symbol, measured loss goes 0.115122 (n = 25) → 0.0335235 (n = 200), while the closed form goes 0.0354978 → 2.5e-12.

2. The control variable is the pool window, not the fraction

Here is the one that should change how you provision.

Hold the per-generation fresh-data rate fixed and vary only w, the number of generations of history your training pool averages over. w = 1 is replacement; w > 1 is accumulation — which is what most real pipelines do.

The stability boundary falls monotonically in 4 of 4 configurations, by two to three orders of magnitude, from changing nothing but how much history the pool carries.

A pipeline that accumulates its pool is not "the same loop with more data". At an identical fresh-data budget per generation it is between two and three orders of magnitude closer to collapse. Conversely, a fraction calibrated under a replacement assumption under-provisions any accumulating pipeline — badly.

The mechanism was stated as a prediction before it was measured: pooling turns exponential mean reversion of rate λ into a moving-average relaxation, whose slowest mode is the eigenvalue nearest 1 of a degree-w companion matrix.

import numpy as np

def slowest_ma_mode(lam, w):
    """Slowest relaxation rate of a w-generation moving average under
    exponential reversion of rate lam. Predicted before measurement."""
    c = np.zeros(w)
    c[0] = 1.0 - lam
    c[w - 1] = -lam
    A = np.vstack([c, np.eye(w - 1, w)])      # companion matrix
    eig = np.linalg.eigvals(A)
    return 1.0 - np.max(np.abs(eig))

Measured rates match that prediction within 2.62 % across 12 cells spanning 82.9×, while the rival replacement law is wrong by up to 32.8×.

3. "Stable" is two questions, and prevention is the expensive one

This is the result I would most want a practitioner to read. Take one loop and compute the fresh-data rate it needs under two criteria that are both natural, both used in the literature, and both computed on the same runs:

• Prevention (λ_stationary): largest λ at which the worst symbol's stationary absence rate stays ≤ 1 %. "The loop stays intact."
• Recovery (λ_heal): smallest λ at which a fully collapsed loop returns to full support within the horizon with probability ≥ 0.5. "The loop can be rescued."

Both located by bisection on a log scale at 1.4e-06 decades resolution:

Prevention is the expensive criterion and recovery is the cheap one. That is the opposite of what most people guess. A loop that has already collapsed is easier to rescue than a healthy loop is to protect, because "every symbol present at every generation" is a far stronger demand than "every symbol comes back eventually".

Note the two-hot w = 1 cell: the prevention criterion is unsatisfiable inside the parameter domain — it reads exactly 1.0. It is reported as a domain ceiling, not as a number, and excluded from the 13.4×–27.6× range. That is a result, not a failed cell, and it is the kind of thing a table without a footnote would have hidden.

One more from the recovery side: more samples per generation help recovery at fixed rate. The two-hot loop goes from P(heal) = 0 at n = 50 to 0.803 at n = 200, at λ = 0.005. Recovery is also not smooth — it is effectively 0/1 in the two-hot n = 50 row, moving from 0 to 0.351 between λ = 0.005 and λ = 0.02.

4. There is no point of no return — only a horizon you will not wait out

The paper measures first-passage times of the same event ("all K symbols present") from both sides: T_loss from a healthy start, T_recov from a fully collapsed one. A ratio above 1 is hysteresis.

That last row is why the censoring fraction is printed beside every number: at n = 50, λ = 0.005, the two-hot loop leaves full support after a median of 2.005 generations, and not one of 200 collapsed replicates returned within the horizon. T_recov is undefined, not large — and "undefined" is the honest word.

And note the 192× row: 0.965 of the recovery-side replicates were censored, so that median is taken over the minority that did recover. The true ratio is larger than the number printed.

The loop is ergodic for every λ > 0. Every state communicates with every other, so there is no point of no return in the strict sense — only a finite-horizon irreversibility whose horizon is set by λ and the pool window. Saying "collapse is irreversible" without naming a horizon is a claim about your time budget, not about the process.

5. The tempting shortcut fails a held-out test

You might hope to replace the boundary computation with one mechanism quantity. Three candidates were fitted to the boundary on one distribution and scored on the other:

• moving-average relaxation rate: misses by a factor of 164.1
• signal-to-noise of the absence indicator: misses by 39.0 % (uniform) and 38.2 % (two-hot)

All three do vary across cells — the relaxation rate spans 82.9×, the stationary variance 0.0011–0.01 (9.24×), the SNR 1.2–2.1 — but none is proportional to λ*. The boundary has to be measured, not proxied.

6. What I would ask for instead

Three things, every time a fresh-data fraction appears in a paper or a design doc:

1. Which criterion — intact support, or recoverability? They differ by 13.4×–27.6× here.
2. Which protocol — replacement or accumulation, and what is w? They differ by up to 0.00101×.
3. Which statistic — the folk lower bound or the measured conditional rate? They differ by a median 0.08437.

A number without those three is under-specified, and that, not sloppiness, is why the field's quotes range from 1 % to 30 %.

The pre-registration, reported honestly

The paper registered three prior beliefs before running anything. Outcomes:

One confirmation, one refutation, and one refutation whose sign is reversed. The reversal was only visible because both criteria were computed on the same loop — which is the argument for computing both, always.

Reproducing it

The package is a committed artefact set: three instruments, a 45-check validation suite tied to the manuscript's claims, a reference layer, and a build step that reads every number in the prose out of a committed *_results.json (nothing is typed in). I ran it from a clean checkout of the paper's own branch:

VALIDATE 45/45          RESULT: PASS
SELFTEST 14/14 plants caught
CHECKSUMS [HARD] 30/30 files match the committed record
wall-clock: 77 s        RESULT: PASS

The 14 plants are the part that matters: the validation suite was itself tested by corrupting each claim and confirming the suite goes red. A suite that has only ever passed is a suite nobody has tested.

Limits stated in the paper and worth repeating: this is a multinomial re-sampling loop over K = 8 symbols, not a neural network — what transfers is the structure (the Jensen bound, the moving average, the criterion dependence), not any specific constant. The thresholds (1 % absence, 0.5 healing probability) are conventions; the result depends on the criteria being different objects, not on the particular values. And the artefact is byte-stable on one build (CPython 3.9.6, numpy 2.0.2); every check is a tolerance comparison, because CPython 3.12 changed the summation algorithm.

The paper is Prevention Is Not Cure: What Fresh-Data Rate a Self-Consuming Training Loop Needs to Stay Stable, and What It Needs to Recover — issue #122 of SILICON SCIENCE: Computer Science, published under CC BY 4.0 with its full reproduction package: github.com/argszero/silicon-science-cs/tree/main/papers/issue-122.

If you maintain a self-consuming pipeline: what is your w, and which of the two criteria are you provisioning for? That second question is the one I got wrong for a long time.