數據之牆：為何合成數據只能爭取時間，卻無法提升準確度

For a decade the recipe for a better model was simple: more data, more compute, more parameters. That recipe is now hitting a wall that is not made of chips. It is made of text.

The stock of high-quality human writing on the open internet is finite, and the largest models have already read most of it. The industry's answer is to generate more text with models themselves — synthetic data — and to mix it into training. This works, for a while. Then it does something subtler and more dangerous: it teaches a model to imitate its own reflection.

Part One: Why the supply of human text runs out

Human text is not scarce in the way oil is scarce. It is scarce in the way clean water is scarce: there is a lot of it, but most of it is not the kind you want to drink. A model trained on the open web learns from forum arguments, SEO spam, and duplicated boilerplate as well as from books and papers. The high-quality subset — carefully edited prose, verified facts, reasoned argument — is a small fraction of the whole, and it grows slowly because it takes humans time to write.

Demand, meanwhile, has grown far faster. Each generation of frontier models wants several times more tokens than the last. The gap between what training wants and what human writing can supply is now wide enough that the industry cannot close it by scraping harder. The data wall is not a shortage of words but a shortage of words worth learning from, and that is a much harder problem to solve.

Part Two: How synthetic data fills the gap

Synthetic data is text produced by a model and then used to train another model, or the same one. It is cheap, fast, and unlimited. You can generate a million examples of a reasoning task overnight, filter them for correctness, and train on the survivors.

The technique works best when it is grounded. If a model generates candidate answers and a verifier — a checker, a test suite, a simulator — keeps only the correct ones, the resulting data is genuinely useful. This is how synthetic data has improved coding assistants and math tutors: the verifier does the quality control that the generator cannot.

It works worst when the generator is also the judge. A model asked to produce training data and then to grade it will reward its own habits, including its own errors. The errors compound across generations, and the model drifts toward a narrower, more confident version of what it already believed.

Part Three: The mechanism behind model collapse

The failure mode has a name: model collapse. It is not a dramatic crash but a slow narrowing. When a model trains on its own output, rare cases disappear first. The long tail of unusual phrasing, minority viewpoints, and edge-case reasoning gets pruned because the model rarely generates it and the filter rarely keeps it. Each generation is a little more average, a little more self-similar.

The fix is to keep a floor of real data in the mix. Studies of data mixing suggest that a modest share of synthetic data — roughly a fifth to a third — improves performance, while pushing past half begins to erode reasoning and factual recall. The exact threshold depends on the task and the verifier, but the shape is consistent: synthetic data is a supplement, not a substitute.

This is why the most credible labs talk about "data mixtures" rather than "synthetic data." The art is in the ratio and the filtering, not in the generation. A model that generates endlessly and filters carelessly will get cheaper to train and worse to use.

When the synthetic share crosses roughly half and the filter is the model itself, each generation loses the rare cases first, and the long tail of reasoning quietly disappears before anyone notices the benchmark has stopped moving.

Part Four: The market and the street

A market estimate clarifies the stakes. If synthetic data lets labs hold training costs flat while compute grows, the near-term effect is to delay the point at which data becomes the binding constraint. Over five years, the synthetic-data tooling market — generation, verification, and curation — could plausibly reach the low tens of billions of dollars, because every lab and every enterprise fine-tuning a model needs it.

The social effect is more intimate. Synthetic data is already inside the tools people use daily: the coding assistant that suggests a fix, the tutor that explains a proof, the customer-service bot that answers a billing question. As real data runs thin, the quality of these tools will depend less on how much text was scraped and more on how well the synthetic mix was verified. A household that relies on an assistant for homework help will feel the difference as a slow drift in reliability rather than a sudden outage.

There is a business-model consequence too. Companies that own clean, verified, proprietary data — hospitals, banks, manufacturers with sensor logs — become more valuable, because their data cannot be scraped and cannot be synthesised faithfully. The data wall pushes value from the public web toward private archives.

What is actually transferable

The lesson is not "synthetic data is bad." It is that synthetic data is a tool with a narrow safe range, and the range is set by the quality of the verifier, not the size of the generator. A team that generates a lot and checks a little is buying speed it will pay for later.

For practitioners, the practical rule is to treat real data as the anchor and synthetic data as ballast. Keep a verified floor of human-written examples, filter synthetic output with an independent check, and measure accuracy on a held-out real set, not on the model's own generations. The teams that do this will keep improving while others plateau.

A model trained mostly on its own output does not get smarter; it gets more certain, and certainty without new evidence is just a slower way to be wrong.

The data wall is real. The way through it is not to generate more, but to verify better.

SOL: HCqPmPJM3NVtS6YrmVaTZYnuP8XZ2PFgQZKT94VK1LkS

If this saved you time, a tip is appreciated.