Innovation Wormhole 回溯案例研究與預先註冊試點方案

Every article in the Innovation Wormhole series stands on its own — no prior reading is required. For context: Part 1 introduced the Innovation Wormhole — the idea that AI should transfer proven mechanisms between industries instead of reinventing them. Later articles defined the protocol (cards, scores, evidence gates), the measurement discipline (calibration against realized outcomes), and the reference architecture (the Transfer Stack). This closing piece is the validation one.

Every technical series faces the same test in its final installment: did any of this survive contact with reality?

This article answers that question honestly, in two parts. First, a retrospective case study — a backtest that replays documented, historical cross-domain transfers through the Transfer Stack and checks whether the protocol's scoring logic would have said the right things. Second, a preregistered pilot protocol — the evaluation design a real deployment must commit to before seeing results, borrowed from the way clinical trials protect themselves from wishful thinking.

The Honest Version of a Case Study

A dishonest case study claims production results that cannot be checked. The evidence principle this series has held from the start — hallucinated innovation is worthless — applies to the series itself.

So, stated plainly: this article does not report results from a production deployment. It reports something that is both more honest and, in an important sense, more useful: a backtest against public history, where the outcomes are already known and independently documented.

The method mirrors financial backtesting: run the pipeline against historical cases where the ground truth is a matter of public record, and ask whether the scoring function would have separated the winners from the losers using only information available before the outcome. History has already run the experiments; a backtest converts them into labeled data.

One warning carried over from finance: backtests overfit, and survivorship bias is severe — the literature celebrates Velcro and forgets the thousand hook-and-loop imitators that went nowhere. Every claim below is therefore scoped to these three transfers, not to a distribution.

The Backtest Design

Three documented transfers form the cohort — one success, one partial success, one failure — chosen because their outcomes are public, well-sourced, and instructive in different ways:

For each transfer, the reconstruction below shows what the Transfer Stack is specified to compute — the cards, the score decomposition, the assumption checks. These are reconstructions against the protocol's definitions, not logs from a production run, and the score decompositions are illustrative walkthroughs of the arithmetic the stack performs. The historical outcomes, by contrast, are documented facts.

Reconstruction 1: The Magnetron That Cooked Dinner

The cavity magnetron, the radar tube that helped win the Second World War, was transferred into the kitchen by Raytheon's Percy Spencer, who noticed a melting candy bar near an active radar set, investigated, and filed the patent for the first microwave oven in 1945 (Percy Spencer, Wikipedia). The mechanism — dielectric heating of polar molecules by microwave radiation — transferred perfectly. The economics did not, at first: the 1947 Radarange stood six feet tall, weighed about 750 pounds, and cost around $5,000 — roughly $72,000 in today's money (Microwave oven, Wikipedia).

What the stack's scoring would have surfaced, at two points in time:

The teaching point: the mechanism never changed between 1947 and 1967. What changed was R_econ. A scoring function that separates mechanism quality from economic readiness — which is exactly what the TFS decomposition exists to do — would have said "the physics is right, the market is twenty years early," and it would have been right.

Reconstruction 2: When the Immune System Met the Firewall

The second reconstruction is the one this series has used as a running example, because its history is genuinely instructive. In 1994, Forrest, Perelson, Allen, and Cherukuri published negative-selection algorithms — generating detector sets filtered against "self" samples — inspired directly by the immune system's T-cell maturation. The transfer into intrusion detection followed almost immediately, and the field of artificial immune systems grew substantially.

The documented reality: the mechanism transferred conceptually and produced a research field, but the assumption set did not fully survive contact with production networks. Three assumptions held (learning "self" without attack examples, anomaly alerting, distribution). Two broke in practice: detector generation does not scale cleanly to high-dimensional, heterogeneous traffic, and "self" drifts as legitimate infrastructure changes. The result was partial success — real deployments and a lasting research lineage, but no dominant commercial product.

The stack's post-hoc arithmetic, against the definitions from the measurement article: ASR = 3/5 = 0.6, with the surviving assumptions clustered in the functional category and the broken ones clustered in the scale and stationarity category. That is precisely the kind of pattern a calibration corpus of Realized Transfer Records is designed to accumulate: assumption categories that break together across transfers. After enough RTRs, the scoring function can price scale-risk assumptions at a premium — which is the whole reason the series insisted on assumption-level bookkeeping rather than verdict-level labels.

Reconstruction 3: The Five-Billion-Dollar R_econ

Iridium was, mechanically, a triumph: 66 satellites in low Earth orbit, global coverage including the poles, operational service from 1998 (Iridium satellite constellation, Wikipedia). The transfer — space-qualified telecommunications engineering applied to global consumer mobility — worked exactly as engineered.

Commercially, it was a catastrophe. By 1999 the venture was in Chapter 11; the constellation's assets were later sold out of bankruptcy for $25 million, erasing more than $4 billion in debt, and the service survived only after the U.S. government stepped in with a $72 million contract (Iridium Communications, Wikipedia). A contemporary New York Times analysis quoted a wireless market analyst's verdict on the "one number anywhere" proposition: "expensive… There never was a viable market" (Iridium satellite constellation, Wikipedia).

Run through the stack's decomposition:

• S_struct, S_func, S_causal — all high. The engineering was flawless, which is exactly why this case is the crucial one: every mechanism-level signal said yes.
• R_ctx — moderate but manageable.
• R_econ — catastrophic and, in hindsight, visible in advance: handset price points, per-minute costs, and the timing of terrestrial cellular build-out were all public information before 1998.

A system that scores mechanism quality and economic friction in one number would have buried the warning. A system that decomposes them — which is the entire argument for the TFS decomposition over a single learned score — would have printed the risk in bold: flawless mechanism, absent market. The failed transfer is the validation case the other two cannot be.

What a Backtest Can and Cannot Prove

Stated explicitly, in both directions:

The backtest supports: that the protocol's decomposition separates mechanism quality from economic readiness; that assumption-level bookkeeping surfaces the categories that actually break (scale, stationarity); that the scoring logic, applied with pre-outcome information only, ranks these three transfers correctly.

The backtest cannot support: production performance claims — throughput, extraction quality, live Precision@k, or calibrated weights fitted on a real corpus. Three cases are an illustration, not a distribution. Those claims require the pilot.

The Preregistered Pilot Protocol

The second half of the honest case study is the deployment design a real organization should commit to in advance — preregistered, the way clinical trials preregister their endpoints, so that success cannot be redefined after the fact.

Scope: one vertical, two unrelated industries, ~500 curated Mechanism Cards, 90 days, per the Transfer Stack's build order.

Preregistered hypotheses:

• H1 (Retrieval): For a set of held-out problem cards with expert-validated historical transfers, the dual index surfaces the validated mechanism in the top-k candidates at a rate meaningfully above a whole-text RAG baseline. Baseline and k are declared before the pilot starts.
• H2 (Calibration): The fitted TFS weights are stable under walk-forward splits — the sign and rough magnitude of w_s, w_f, w_c do not flip between folds. Unstable weights mean the calibration corpus is too small to support scoring claims.
• H3 (Assumptions): Assumption-level predictions (which assumptions survive in the target context) agree with expert post-hoc judgment more often than verdict-level predictions (does the transfer work?) — testing whether the bookkeeping granularity is worth its cost.

Preregistered go/no-go gates:

• Gate 1 (Evidence integrity): ≥ 98% of cards pass citation resolution; sampled human verification finds no fabricated evidence. Failure stops the pilot — nothing downstream is meaningful.
• Gate 2 (Retrieval): H1 met on the held-out set. Failure means the index layer regressed toward RAG and must be fixed before calibration.
• Gate 3 (Honest scoring): H2 met, and calibration error reported with weights published. Failure means the system ships with heuristic scores, labeled as such — not silently.

What is deliberately absent: any gate phrased as "the AI found a great idea." The pilot measures the instrument, not the poetry. A single genuinely great transfer discovered along the way is a welcome bonus, not an endpoint.

Authorization Boundaries

The standing rule closes the series unchanged, and the backtest adds a specific instance of it:

• The backtest is not a deployment. Reconstructions validate scoring logic, not organizational readiness.
• Preregistration is a human commitment. No AI system may revise its own gates, redefine its endpoints, or re-score its holdout after seeing results.
• Failed gates are results. A pilot that stops at Gate 1 with "extraction hallucinates more than we hoped" has produced knowledge. The audit log exists so that stopping is visible and respectable.

Technical Challenges and Open Problems

• Cohort scale. Three reconstructions illustrate; a calibration corpus needs hundreds of RTRs, and the scarce ones (documented failures) are the valuable ones.
• Pre-outcome information discipline. The magnetron backtest used 1947 prices, not 1967 hindsight. Enforcing that discipline automatically — pinning every feature to sources dated before the outcome — remains an unsolved data-engineering problem.
• Outcome grading. "Partial success" dominated both real transfers studied here. Graded outcomes need metrics that do not collapse them into binaries.
• Survivorship bias in corpora. Any public-collection strategy over-collects successes; failure harvesting (post-mortems, abandoned patents, shut-down pilots) must be an explicit acquisition stream.

Why This Matters

The series closes where it opened, with a claim about innovation: most breakthrough ideas are successful mechanisms transferred from one domain into another — and the hard part is not inventing, but finding where the answer already exists.

What the five parts add up to is a checkable version of that claim: an idea (transfer mechanisms, don't reinvent them), a protocol (cards, scores, evidence, gates), a measurement discipline (calibration against realized outcomes), an architecture (the Transfer Stack), and now a validation method (backtests plus preregistered pilots).

The magnetron waited twenty years for its economic window. The immune-system analogy broke on scale, not concept. Iridium failed with flawless engineering. Each of those facts was knowable in advance to anyone forced to decompose mechanism from market — which is the entire proposition of this series in one sentence.

The next step is public: assemble Realized Transfer Records — especially the failures — and publish the corpus. If you know a documented cross-domain transfer, success or failure, it belongs in the benchmark. I'd love to hear about it in the comments.

References

1. Percy Spencer — inventor of the microwave oven — Wikipedia
2. Microwave oven — Radarange history and pricing — Wikipedia
3. Iridium Communications — bankruptcy and restructuring history — Wikipedia
4. Iridium satellite constellation — system history and market analysis — Wikipedia
5. Velcro Companies — hook-and-loop transfer history — Wikipedia
6. Forrest, S., Perelson, A. S., Allen, L., & Cherukuri, R. (1994). Self-Nonself Discrimination in a Computer. Proceedings of IEEE Symposium on Research in Security and Privacy — the founding negative-selection paper (cited textually; the assumption analysis above is drawn from the subsequent artificial-immune-systems literature).
7. Scideator — Human-LLM Compound System for Scientific Ideation through Facet Recombination and Novelty Evaluation — arXiv:2409.14634
8. Accelerating Innovation Through Analogy Mining — arXiv:1706.05585
9. Cross-Industry Innovation Does Not Happen by Chance: The Role of Facilitation — Springer Professional
10. Part 1 of this series — Building an AI-Powered Innovation Wormhole

created by Seyed Alireza Alhosseini Almodarresieh
https://almodarresieh.github.io/