Revenant 評估 npm 套件沉寂風險，TabPFN-3.5 回測 ROC AUC 為 0.8893

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

picocolors is a tiny library that colours terminal output. It sits in the dependency tree of an enormous slice of the JavaScript ecosystem, and as far as I can tell there is nothing wrong with it.

My model gave it a 0.900 probability of having gone quiet.

The model was right, in the narrow sense I had defined. And that is the most useful thing this project taught me, so I'm going to spend most of this post on why.

What I Built

Revenant reads a package-lock.json and, for every npm dependency in it, estimates the probability that the package has gone quiet: published no new version for twelve months.

Every project carries ghosts. Packages that still run in production, still resolve on every install, and whose last release was years ago. Nobody notices until a vulnerability lands and there is nobody left to ship the fix. Tools can already tell you a package is stale. I wanted to know whether you can tell which ones are about to be, from nothing but their release history.

So I built two things:

1. A backtest. Take the npm ecosystem as it stood on 1 January 2024. Using only what was knowable on that day, predict which packages would publish nothing at all during 2024. Then check.
2. A scorer. Point the trained model at a real lockfile and get a ranked list, with each package's strongest signal and an explicit refusal for packages too young or too thin to judge.

The prediction is done entirely by TabPFN-3.5, an open-weight tabular foundation model, running locally. Gemma has one small job — turning an already-computed row into a sentence — and is never allowed to produce a number.

A note on the word "dormant." Twelve months without a release is not proof that a package is abandoned. Some packages are simply finished. I knew that going in and wrote it into the spec before I started. I did not know how much it would matter. It turned out to be the most important result in the project.

TL;DR

• 3,803 npm packages, labelled from real 2024 release history. 39.52% went quiet.
• On a held-out test set of 1,141 packages, TabPFN reached ROC AUC 0.8893 [95% CI 0.8715, 0.9077], against 0.8380 for logistic regression and 0.5 for guessing the base rate.
• On the metric that matches what the tool actually does — how many quiet packages land in its top 10% — TabPFN and logistic regression tied.
• Remove the most obvious feature (days since the last release) and TabPFN barely notices. Logistic regression falls apart.
• The model's most confident mistakes all say the same thing: a release history cannot tell a finished package from an abandoned one, and it cannot see a project being stopped.
• My first user asked whether a flagged package ships to production or only runs in dev tooling. I built that the same morning: on my own demo lockfile, 341 of 429 scored packages turned out to be dev-only.
• Everything — cutoff, label, sample, features, metrics, and the refusal rule — was committed to a public repo before a single package was downloaded.

Demo

Revenant is a command-line tool, not a hosted app. I spent the build on the backtest rather than a UI, and I'd make the same trade again.

There's no video here: my screen recorder didn't capture the terminal, and the scorer takes about seven minutes to run on a laptop CPU, so there wasn't time to rerun it before the deadline. The screenshots below are from the run itself, and the complete output is committed as results/explain_demo.txt, so you can check every line.

The commands:

python src/backtest.py                                      # the pre-registered evaluation
python src/score.py examples/demo-lockfile.json             # score a real lockfile
python src/score.py examples/demo-lockfile.json --explain   # plus Gemma's explanations

On the demo lockfile, one of my own projects, Revenant found 492 unique packages. It scored 429, refused 63 under the pre-registered abstention rule, and failed to fetch 0.

The top of the ranked list. Every probability is printed below the base rate and the model's measured Brier score, so a 0.9 can be read in context. The scope column shows whether each package ships to production or only runs in dev tooling.

The refusals. @alloc/quick-lru has two lifetime releases, and the rule needs five, so Revenant says so instead of guessing.

The rule I set before I looked

Before I fetched a single registry document, I wrote down exactly what I was going to measure and committed it to a public repo:

• Cutoff: 1 January 2024. Every feature is computed as of that instant.
• Label: dormant if the package published no version between 1 January 2024 and 1 January 2025. One window, fixed.
• Sample: random npm packages that were alive at the cutoff — at least one release in 2023, at least five versions, at least two years old.
• Eleven features, all derived from publish timestamps.
• Three models on one identical split, with confidence intervals, a calibration check, and an ablation removing the strongest feature.
• A refusal rule: no score for a package with fewer than five releases or under a year old.

That file is PREREGISTRATION.md, frozen at commit 4403383 at 2026-10-31T09:33:54+05:00 — before any data existed on my machine.

I did this because the easiest way to fool yourself with a model like this is to adjust the question after seeing the answer. Redefine "dormant" as eighteen months instead of twelve, drop an awkward feature, pick the threshold that flatters the result. Any one of those is defensible on its own. All of them together turn a test into a story.

Pre-registering cost me nothing but a file, and it paid off twice. It stopped me moving the goalposts. And the ablation I had promised in advance — without knowing whether it would show anything — produced the most interesting result in the project.

Every later judgement call went into MEMORY.md before the model it affected was fitted: how the threshold would be chosen, how a slightly ambiguous date filter would be read, and when collection would stop.

How I Built It

The data: one field, fully historical

Every npm package has a public registry document, and that document contains a time field: the exact publish timestamp of every version ever released. It is complete and historical. I can reconstruct what any package looked like on 1 January 2024 without guessing.

That one field is the entire basis of the model. I deliberately used nothing else.

The tempting extras would have broken the experiment. Aggregator APIs report today's maintainer counts and issue statistics, which encode what happened after the cutoff. Download counts looked like the obvious way to filter out junk, but the npm downloads API only serves roughly the last eighteen months of history. Asking for 2023 silently returns recent data instead. And filtering by today's downloads would select for packages that survived — exactly the thing I was trying to predict.

I also wanted deprecation in the label, but the registry tells you a version is deprecated, not when it was deprecated. A label I could not date as of 2025 was a label I could not use, so I dropped it and said so in the spec.

Sampling: 4.5 million names, 3,803 rows

I enumerated every package name from npm's replication endpoint — 4,552,046 names in about eleven minutes. The registry reported six more by the time I finished, because it changes while you read it.

From a permutation of that list seeded at 20261031, I fetched 140,000 registry documents. 139,726 came back; 274 were 404s; 28 transport errors all recovered on retry.

Only 3,803 packages — about 2.7% — passed the three frame filters. Most of npm is one-version experiments and abandoned stubs. That attrition worried me at first. Then I realised it cuts the right way: the survivors look much more like the packages that actually appear in lockfiles.

Eleven features, all cutoff-safe

Leakage is enforced in code, not in comments. Every feature function raises an error if it is handed a timestamp at or after the cutoff. A test builds a synthetic package whose only post-cutoff activity is a 2024 release and asserts its features are identical with and without it.

One small honest fix: gap_trend divides the recent gap by the earlier gap, and some packages publish several versions within the same minute, which makes the denominator zero. I add one day to both sides. That decision was recorded before any model was fitted.

Three models, one split

A stratified 70/30 split: 2,662 packages to train, 1,141 to test.

• TabPFN-3.5, running locally. It is a transformer pre-trained on synthetic tabular problems, so it needs no hyperparameter search. You hand it the training table and it predicts.
• Logistic regression on standardised features — the boring, sensible baseline anyone would try first.
• The base rate — every package gets 0.3952. Anything worth using has to beat this.

The decision threshold was chosen by maximising Youden's J on five-fold out-of-fold predictions inside the training split only. The test set was never consulted.

Results

Confidence intervals are percentile bootstrap over the test split, 2,000 resamples.

Where the gap lives: between roughly 5% and 30% false positives, TabPFN finds noticeably more quiet packages. Past about 60%, the two curves converge.

TabPFN wins on AUC, and the win holds up. A paired bootstrap on the difference gives 0.0514 [0.0365, 0.0666] — an interval that doesn't touch zero. (That paired test was not in my pre-registration, so I'm labelling it exploratory.)

At their chosen thresholds, the two models make different trades. TabPFN flagged 519 packages and was right about 371 of them. Logistic regression flagged 606 and was right about 386. Logistic regression catches fifteen more quiet packages, at the cost of seventy-two more false alarms.

Then it tied

Here is the number I care most about, and it isn't AUC.

The thing Revenant actually does is put the riskiest packages at the top of a list. So the question that matters is: of the packages in the top 10%, how many really went quiet, compared with chance?

TabPFN: 2.244 times the base rate. Logistic regression: 2.200. The confidence intervals overlap almost entirely.

On the job the tool performs, the foundation model and the boring baseline are indistinguishable. If I had only reported AUC, I could have hidden that. It's in my pre-registration, so it's in this post.

Where TabPFN does earn its place

When TabPFN says 0.55, roughly 55% of those packages went quiet. Logistic regression is less trustworthy in the middle of the range, where most real decisions sit.

TabPFN's Brier score is 0.1327 against 0.1624. Its probabilities mean what they say.

For a tool that hands you a number and asks you to act on it, that matters more than ranking. A probability you can trust is one you can set a policy on: review anything above 0.7. A probability you can't trust is decoration.

What it was actually reading

The ablation I committed to in advance: refit everything without days_since_last_release, the single most obvious signal.

Logistic regression leaned on that one feature. Take it away and it loses more than three times as much.

TabPFN barely notices. It was reading the shape of the release history — the rhythm, the slowdown, the long silences that came before — and that structure carries nearly the same information without the headline number.

That's the clearest case for a tabular foundation model in the whole run, and it exists only because I promised to look before I knew what I would find.

Where it was most confidently wrong

This is the section that changed how I read every other number in this post.

The three packages TabPFN was most sure would go quiet, that didn't:

Two of the three are TypeScript type definitions. @types packages are republished by DefinitelyTyped's tooling when something upstream changes, not on a maintainer's rhythm. Long silences are normal for them. A long silence followed by a release is exactly what "finished, but maintained" looks like.

The three packages TabPFN was most sure would keep going, that went quiet:

These were busy. Two dozen releases a year, the last one less than a week before the cutoff. Then they stopped completely. Whatever happened to them happened somewhere their release history couldn't see.

Put the two lists side by side and the lesson is the same from both directions. A release cadence can't tell a package that is finished from a package that is abandoned, and it can't see a project being stopped. The first produces confident false alarms. The second produces confident misses.

Then I ran the scorer on a real lockfile, and the top of the list was which-boxed-primitive at 0.918, is-shared-array-buffer at 0.912, and picocolors at 0.900.

These are small, single-purpose packages that do one thing well and have nothing left to change. picocolors hadn't released anything in 745 days when I ran it: nine versions over five years, then silence. Under my definition, it really has gone quiet. Under any common-sense definition, all three are fine.

That is not the model failing. It is the model faithfully answering the question I asked — and discovering, at scale, that my question has a blind spot. The weakest part of Revenant is not TabPFN. It is my label.

I wrote that warning into the pre-registration on day one. I did not expect the data to agree with it this loudly.

The Spirit of It

I asked a friend to let me run Revenant on one of her own projects. She agreed on the condition that I keep her anonymous, and that she'd tell me honestly what she thought.

Revenant scored 440 of her dependencies and refused 63 under the abstention rule.

Its top three were which-boxed-primitive at 0.917, is-shared-array-buffer at 0.913, and picocolors at 0.900. Those are the same three packages, in the same order, that topped my own lockfile, in a project she built independently. (The scores differ slightly between the two runs because they were taken at different times, and the days-since-last-release counter keeps going up.)

Small, finished packages like these end up in a huge share of JavaScript projects, so they will sit at the top of nearly any lockfile you point Revenant at. The label's blind spot doesn't just show up once in a backtest table. It shows up first thing in every real project.

I sent her the top pick and asked what she made of it. Her reply:

Honestly no, I had no idea that this package was in my project. So yes, it's useful, it showed me a blindspot I didn't know I had, but my one suggestion is that it should show whether a flagged package actually ships in the live app or only runs in dev tools. When I checked which-boxed-primitive, it turned out to be a tiny helper used only during development. It probably hasn't been updated because it's finished, not because it's abandoned. Knowing that up front would tell me right away whether to act or move on, and I'd trust the rest of the list more.

She reached the same conclusion as the backtest, from the opposite end. The backtest found it in 1,141 held-out packages and a table of confident mistakes. She found it by looking up one package. Finished, not abandoned.

Her suggestion is also better than mine. I had been thinking about how to fix the label. She was thinking about what a person needs in order to act on a score. A quiet package that only runs in dev tooling is a curiosity. A quiet package that ships to users is a liability. That distinction costs nothing to compute, and it decides whether a 0.917 means anything to the person reading it.

So I built it that morning

npm lockfiles already record this. Every entry carries "dev": true if only development tooling depends on it. Revenant now reads that flag and adds a scope column, dev or prod, to every scored package. A package counts as dev-only only if every lockfile entry for it is marked dev. Entries npm marks devOptional are counted as production, because they can be pulled in as optional dependencies of production code.

On my own demo lockfile, the result changed how I read the whole list:

Of 429 scored packages, 341 are dev-only. Only 88 ship to production.

which-boxed-primitive, my top-ranked package as well as hers, is dev. So is escalade, and so are most of the finished micro-packages crowding the top of the list. The first prod package in the ranking is picocolors, at 0.898. That's one that actually reaches users, and the one in this post's title.

The model didn't change. The probabilities didn't change. But a list of 429 scores became a list of 88 that matter, plus 341 the reader can check later, all because a first user asked one practical question.

Gemma, kept on a short leash

A probability and a feature name aren't much use to someone who isn't a data scientist. So there's a final, optional step: a local Gemma model, running through Ollama, turns one scored row into one plain sentence.

It is deliberately constrained. Gemma never computes, compares, ranks or rounds anything. Python formats every number into a fixed template first, and Gemma's only job is the wording. It never sees free text from a registry document, so a package README can't steer it.

A guard checks this mechanically: every number in Gemma's output must be one it was given. If it ever invents a figure, the sentence is rejected. If Ollama isn't running, it falls back to the plain Python template.

Here is what Gemma, running locally, wrote for picocolors, word for word:

picocolors has a model probability of 0.90 of publishing no new version in the next 12 months, with its last release 745 days ago, 0 releases in the past 12 months, and 9 releases over 5.1 years.

Every number in that sentence came from Python, and Gemma's only job was the wording. That's all I want from a language model here. The guard passed it because there was nothing new to catch.

Gemma's sentences for the five top-ranked packages, each labelled with where it came from.

It's wired into the scorer as an opt-in flag. python src/score.py <lockfile> --explain prints a sentence for each of the top five packages and labels each one with where it came from, Gemma or the template fallback. On the demo lockfile, all five came from Gemma. The full output is committed in results/explain_demo.txt.

Code

Abeera81 / revenant

Revenant: scores npm dependencies' probability of going dormant

Revenant

Scores each npm dependency in a package-lock.json by its probability of going dormant.

A package is labelled dormant if it went quiet: it published no new version during the fixed label window defined in PREREGISTRATION.md.

On commit dates

All commits were made before the Oct 31, 11:59 PM PDT deadline. They display as Nov 1 because every commit timestamp in this repo carries a +05:00 (Pakistan) offset; subtract 12 hours for Pacific time. For example 2026-11-01T09:47:43+05:00 is 2026-10-31T21:47:43 PDT.

Usage

python src/score.py path/to/package-lock.json --explain (ranked scores with a prod/dev column; --explain adds a one-sentence explanation for the top 5 via local Gemma, or the template fallback)

Results

Rows 3803 (train 2662, test 1141), stratified 70/30, random_state=20261031. Base rate (dormant): all 0.3952, train 0.3952, test 0.3953. CIs: percentile bootstrap over the test split, 2000 resamples.

…

• PREREGISTRATION.md — the frozen spec
• MEMORY.md — every decision, and when it was made
• results/backtest.md — every number in this post
• src/ — collector, features, labels, backtest, scorer, verbaliser

I built this in two coding agents. The pre-registration and the registry collector were written with GitHub Copilot's agent. When I hit its monthly quota partway through, I moved to Claude Code. The handover took a single message, because the first agent had kept its state in AGENTS.md and MEMORY.md instead of in a chat window.

Why Does Open Innovation Matter?

Three things in this project only worked because they were open.

The data had to be open to be honest. The backtest exists because npm publishes every version's timestamp, publicly and permanently, so the ecosystem's past can be rebuilt exactly as it stood. Openness also showed me precisely where the data ran out: the downloads API stops at eighteen months, and deprecations carry no date. I could see those gaps and design around them instead of trusting a number I couldn't check.

The model had to run on my machine. A lockfile looks harmless, but it can name internal packages, private registries, and the client a project was built for. My friend could hand me hers because it would never leave my laptop. With TabPFN running locally, nobody's dependency graph is uploaded anywhere.

A promise needs something fixed to promise about. A pre-registration commits to a procedure. With open weights, the model I tested is the model you can download and rerun against my frozen spec. A hosted model can change behind its API, and there's nothing to pin.

Limitations, plainly

• The label measures silence, not death. Twelve months without a release is a proxy. The @types false alarms and picocolors show exactly how it misleads.
• Training data and real lockfiles are different populations. The model learned on random npm packages that passed my filters. Real lockfiles are curated and lean towards small, stable utilities — exactly the packages my label misreads.
• Deprecation is excluded, because the registry doesn't record when it happened.
• The refusal rule never fires in the backtest. The sample already requires five versions and two years of age, so every backtest package is scoreable by construction. Refusals are only measured on real lockfiles.
• One cutoff, one year. I tested 2024 once. I don't know how stable these numbers are across other years.
• "Largest-contributing feature" is an occlusion estimate. It's how much the probability moves when that feature is reset to its median. It's an association, not a cause.
• Commit dates. Every commit carries a +05:00 Pakistan offset, so GitHub shows Nov 1. All were made before the Oct 31, 11:59 PM PDT deadline.

What I'd do next

The fix isn't a better model. It's a better label.

My friend's suggestion is already built. The next fix is the label itself. The obvious step is a third class. Split quiet into finished and abandoned, using signals that tell them apart: open issues nobody answers, security advisories with no fix, dependents migrating away. Those signals are messier and harder to date, which is exactly why I didn't use them under a deadline.

Prize Categories

• Best Use of TabPFN — TabPFN-3.5 does all prediction, locally, inside a pre-registered prediction pipeline evaluated against two baselines.
• Best Use of Gemma — local Gemma through Ollama explains the top-ranked packages in plain English (--explain), under a guard that rejects any number it wasn't given.
• Best Use of GitHub Copilot — the pre-registration and the registry collector were built with Copilot's coding agent.

I set out to build something that could find the ghosts in a dependency tree.

What I built finds the quiet ones, and then tells you, with a number you can trust, that it can't tell a ghost from something at rest.