跳到正文
DEV Community· Nabil A·· 35 分鐘前AI 評分45

Since 升級檢查器對每次升級都判定「未能定論」

Our Upgrade Checker Said "Not Settled" to Every Single Upgrade

AI 導讀

Since 將升級檢查器用於 300 個真實升級 PR,首 455 次判定全為「未能定論」。已完成的 41,298 次版本更新説明讀取中,只有 21.1% 完整讀取;檢查器亦有四項會導致判定未能定論的錯誤。團隊修正了這些錯誤,並保留只有讀取所有版本更新説明後才判定「清晰」的規則。

正文

We built a button that says "this upgrade doesn't break your code." Then we fed it 300 real upgrade pull requests, and for the first 455 moves it said the same thing every time: not settled.

Not "safe." Not "broken." Just a polite shrug, 455 times in a row.

That was the morning of Nov 22, 2026, the day we shipped upgrade receipts at Since. This is the story of why the shrug was partly a bug, partly the truth, and why we kept the truth.

1. What we set out to build

Every developer has stared at a Dependabot-style bump and thought: will this break me? The changelog is long, the tests are green, and you merge on vibes.

Search can't answer that question, because the answer depends on your code. So we built one that does. You give Since your code and the version you want to move to, and it answers with one of three verdicts:

  • Clear: every release between your version and the new one was read in full, and nothing in them reaches your code.
  • Not clear: here is the release, what it changed, and the line in your code it breaks.
  • Not settled: here is what we could not read.

The word "safe" is banned. A clear receipt says "nothing found in what was read," because that is the only honest sentence available.

The rule fits in a few lines:

// The rule behind an upgrade receipt, in miniature.
function receipt(releases) {
  if (releases.some((r) => r.breaksYourCode)) return "not clear";
  if (releases.every((r) => r.read === "full")) return "clear: nothing found in what was read";
  return "not settled: " + releases.filter((r) => r.read !== "full").map((r) => r.version + " read " + r.read).join(", ");
}

console.log(receipt([{ version: "4.20.0", read: "full" }, { version: "4.21.0", read: "full" }]));
console.log(receipt([{ version: "4.20.0", read: "full" }, { version: "4.21.0", read: "in part" }]));
console.log(receipt([{ version: "4.20.0", read: "in part", breaksYourCode: true }]));
clear: nothing found in what was read
not settled: 4.21.0 read in part
not clear

Look at the middle line. One release read only in part, and the whole move is not settled. That one strict word, "every," is the whole product. It is also the whole problem.

2. The first two answers

Launch day, we checked two boring, everyday moves: express 4.19.2 to 4.21.2, and zod 3.23.8 to 3.24.1.

Both came back not settled, each naming the releases we had read only in part.

Fine, we thought. Two popular packages, a few releases with thin notes. Bad luck.

So we measured how often "read in full" happens at all.

3. The number that explains everything

Of the 41,298 release reads we had finished, 21.1% were read in full. By registry it looked like this:

Registry Releases read in full
crates.io 46.9%
PyPI 45.2%
npm 16.4%
Go 10.3%
Terraform, Swift, NuGet, RubyGems, CocoaPods 0%

(Our records, Nov 22, 2026.)

If you want a clear receipt, every release of the move has to be in that first bucket. An upgrade that crosses five npm releases at 16.4% each is, statistically, a coin that keeps landing on its edge.

Why is npm so low? The most common reason a release was only read in part was that its types extend a type from another package: 14,533 of the 14,927 partial npm reads said so. That looked like one fix away from a big jump.

It wasn't. Only 2,135 of them had nothing else missing. The rest were also missing the thing you'd think every release has: notes. No repository listed. A CHANGELOG.md with no section for that version. A releases page that stops after 5 pages. No publish date.

A lot of software ships with no explanation of what changed. We knew that in the abstract. Seeing it as a percentage was different.

4. The replay

Two verdicts on launch day is an anecdote. We wanted a real test, so we collected 300 upgrade-only pull requests from public repositories, npm and PyPI, at most 3 per repository: 260 where CI passed after the bump, and 40 where it broke (the head failed where the base had passed).

We had to throw out a lot to get there: 193 pull requests that changed more than dependencies, 450 from other registries or GitHub Actions bumps, 125 with no CI to compare against, and 53 whose base was already failing.

Then we ran the receipts. And the first 455 moves all said not settled.

5. The four bugs hiding in the shrug

Some of that was the honest 21% at work. Some of it was us.

Bug one: lockfile-only packages could never be cleared. 419 of those first 455 moves were indirect: packages that only the lockfile holds, pulled in by something else. Our code had no path for an indirect move to ever reach "clear." So it never did. Now an indirect move is read through what your packages actually call of it, and it rests on its own releases.

Bug two: "not settled" with no reason. A receipt where every release was read in full could still say not settled, with nothing after the colon, if some other package in the same check was still being read. The receipt was waiting on a stranger. Each move now stands on its own.

Bug three: the false clear we caught before it happened. This is the one that still makes me wince. A lockfile-only pull request sent no manifest, so a package your code declares directly was treated as indirect, and checked against what its dependents call instead of what your code calls. That is exactly how you get a "clear" that isn't. It hadn't fired yet. It was waiting. Now the manifest beside a changed lockfile is always read.

Bug four: one big changelog sank the whole step. A changelog over 2 MB failed its read entirely. 367 steps had failed on it. Now it's read up to the cap, and the rest is named as not read.

That's the part I'd underline. Every one of these bugs pushed toward "not settled," except the third, which pushed toward a wrong "clear." A strict tool fails quiet and safe most of the time. The one bug that could make it loud and wrong is the one worth losing sleep over.

6. After the fixes

On the fixed harness, the first 44 pull requests gave us: 36 moves cleared, 3 whole pull requests clear, and no false clear.

Three out of 44. I'm not going to dress that up.

The ones still not settled split three ways: 135 moves whose releases were still being read, 111 with a release read only in part (mostly packages with native bindings, and JavaScript with no types, where the honest answer is "we'd have to read the package's code"), and 16 where no code imports the package at all.

The full replay isn't finished, so there's no published share yet. I'm telling you the interim numbers because the shape matters more than the total.

7. The temptation

Here's the conversation every team building something like this has, usually with itself.

"What if we counted 'read in part' as good enough when the missing part is just types?"

"What if we cleared a move when the release notes don't mention anything that looks like your API?"

Either one would turn a lot of not settled into clear by tomorrow. Both would make "clear" mean less.

We didn't touch the definition. Instead we went after the reasons releases weren't read in full, at the source:

  • a NuGet package that ships one target framework is now read in full
  • a @types package is read in full from its declarations
  • release notes are found where a repository actually keeps them: odd tag spellings, releases past the first pages, Go modules living in subfolders

After that last change, 8,793 of 43,192 finished reads were in full (20.4%, Nov 22, 6:35 AM). Of the first 31 releases read again, the "no section in the changelog" gap went from 14 to 4.

It's slow. Our public stats for the last 7 days, read this morning, show 2,928 releases read in full out of 15,705 (18.6%): 11,555 in part, 785 unreadable, 437 failed (since.dev stats, Nov 22, 9:34 AM MST). Every receipt we can't settle names what we didn't read.

When this is the wrong call

If you need an answer on every upgrade today, a strict "every release read in full" rule will frustrate you, and a looser tool that says "probably fine" will feel better. For a solo side project with good tests, your CI is honestly a fine receipt.

The strict rule earns its keep when someone downstream trusts the word "clear" without reading further: an agent deciding whether to merge, or a team that blocks merges on it. There, one false clear costs more than a thousand shrugs.

What I took from it

A tool that answers "I don't know" most of the time is only useful if it tells you why, and if its "yes" is worth something. We'd rather ship 3 clear pull requests out of 44 that mean it than 40 that don't.


What's the upgrade you merged because the tests were green, that turned out to break something the tests never touched?

I'm Nabil. I build Since, which tracks what your code depends on and tells you before it breaks.

來源:DEV Community · dev.to