Git 淺層 clone 會讓歷史檢查回傳看似有效的錯誤結果

Originally published at erikhill.dev. The numbers below are checked against the repository they come from.

A shallow clone does not fail your history check. It answers it.

I replaced a check that could not tell it had stopped working. The
replacement could not tell either. The commit landed at 18:44:13 and the run
that caught it started at 18:44:22. Nine seconds.

What happened

I committed a new check at 18:44:13 and the run that caught it failing started
at 18:44:22. Nine seconds. It had passed locally, 282 tests green. In CI it
accused my own commit of secretly editing the results board that every
published number is derived from.

The failure read:

board commits that are neither the probe nor a declared human edit:
  e9f904c5 by egnaro9: Stage the guard-sibling note, with the fifth
  instance and its probe

That commit adds a blog post. It does not touch the board, and it never has.

The check existed because I had just switched off an older one that compared
the board against a second, hosted store. Switched off, not deleted: it is
still in the file, skipping now with its reason stated. Before that it had
spent twenty-four days reporting green without running, because the store it
queried was retired in August and an unreachable store is not the same fact as
a disagreeing one. Fifty-seven CI runs passed while it compared nothing, each
printing 207 passed, 1 skipped.

So the replacement asks git instead, because git is already there and cannot be
retired: has the board been edited outside the run that produced it? Every
change is a commit with an author and a subject. The probe writes
chore: update drift standings. A deliberate human edit says board: and
explains itself. Anything else is a stray. And on my laptop, that worked.

The clone was the problem, and the clone had no error to give

CI checks out at the default depth of 1. My laptop has the full history.

A shallow clone does not make git log -- <path> fail. It does not make it
empty. The single fetched commit has no parent, so by parent list a shallow
boundary and a genuine root commit are the same shape, and the log answered
as though that commit had created the file.

Measured on two clones of the same repository, at the same commit, minutes
apart:

That last row is the whole post. Git reported a 33,403-line insertion that
never happened, attributed it to the person who pushed last, and my check read
that number and did exactly what it was built to do.

Git did tell me. I did not ask.

I wrote the paragraph above claiming git has no way to signal this. That was
wrong, and an adversarial re-check of my own draft caught it before this went
out. The signal exists and it is one flag away:

$ git log --diff-filter=A --decorate -- dashboard/drift_board.json
600d503 (grafted, HEAD -> main, origin/main) A shallow clone does not...

grafted is git saying precisely what I needed, per commit rather than per
repository, so it marks exactly which log lines are untrustworthy. The full
clone prints no such marker on the real creating commit. It survives
--no-decorate being the scripting default, which is the point: nobody adds
--decorate to a query they intend to parse.

So the honest claim is smaller and more useful than the one I made. Git does
not tell you by default, and the field that would tell you is one nobody reads.
The answer was sitting in the output I chose not to ask for.

(Measured on git 2.50.1 only. I have not checked older versions.)

The guard I had already written for this did not fire

I had anticipated the empty case. The check opened with:

assert subjects, "no history for the board; this check did not actually run"

That assertion is correct and it is useless here, because the result was not
empty. It was a list of one. A vacuity guard asks "did I get nothing?" and the
shallow clone's answer is "no, you got something." The failure mode I needed to
survive was never absence. It was a confident, plausible, wrong answer,
which is the only kind a provenance check cannot survive.

And the sibling thirty-six lines below already had the fix

.github/workflows/ci.yml has two jobs that check out the repository. The
second one already carried this, with a comment explaining why:

fetch-depth: 0        # the stamped commit must be reachable

The first one did not. Line 29 and line 65 of the same file, one guarded for
an unrelated reason and one not. I added a check that needed full history to the
job that did not have it, in a file that visibly contained the answer.

How it was found at all, which is the part I got lucky on

Not by reading the email. By a different probe, built earlier the same evening
for that problem, which reports for every scheduled workflow how
many consecutive runs and how many days its verdict has held, and when it was
last green. Run it and one line said:

NEW RED   model-drift   draft.yml   1 runs / 0d, NEVER green in the window

NEVER green in the window means no run it looked at was green. The tool's
own wording is "may never have passed at all" and that hedge is load-bearing:
it reads a fixed number of recent runs, so a long enough red streak is
indistinguishable from a check that has never passed. Here it was the strong
reading, because the workflow had run exactly once. That was a second, unrelated instance of the same family:
a workflow that finds a regression, writes it up, pushes the branch, and then
fails on the final step whose only job is to open a pull request and tell
somebody. The finding existed, complete, in a place nobody would look.

A notification system that fails at the notification step produces silence,
and silence is indistinguishable from having nothing to report.

One thing limited the damage, and it is narrower than I first wrote. The code
records "I have already drafted this finding" into the same commit as the
draft itself. So an unmerged draft leaves the main ledger untouched and the
finding gets offered again on the next run. Had the ledger been written
separately, the run would have marked the regression seen while the only copy
sat on an abandoned branch, and it would never have been raised again.

That is a re-offer guarantee, not a save. The draft itself is gone: I deleted
the branch, and the commit survives only as an unreachable SHA waiting on
garbage collection. I kept a bundle, which is me being careful rather than the
system being safe. And the workflow's own pull request body told the reader the
opposite, that closing the PR records the finding as seen and it will not be
offered again, which was false for exactly the same reason. I fixed that text
while writing this.

None of that is a design I can claim credit for foreseeing. I noticed it while
checking whether I had lost anything.

What I did about it

Both layers, because either one alone rots:

The workflow now fetches full history. That is the one-line fix and it is the
less important one, because a workflow file is a different file and will drift
again.

The check now refuses to answer from a clone that cannot answer. It runs the
log, asks git rev-parse --is-shallow-repository, and throws the answer away
with the remedy in the message rather than reporting a fabricated stray.
Refusing before the result is USED is the part that matters; refusing before it
is fetched would be tidier and buys nothing.

The predicate is a plain function taking the flag and the commit count, so it
can be handed the shallow case directly, which a real clone cannot be made to
do inside a test run. The case that bit me is now
test_a_shallow_clone_is_refused_even_though_it_answers: shallow with one
commit must be refused, even though one passes every vacuity check ever
written.

Proof on a single clone: shallow, two failures naming fetch-depth. Then
git fetch --unshallow, same clone, same file, thirteen passed. Five of five
mutations caught.

What I might have wrong

The general claim I want to make is "check the integrity of your environment,
not just your reading of it," and I am not sure that generalizes without
becoming unfalsifiable. Every check runs in an environment. You cannot verify
all of it. I can defend the specific version: if your instrument reads
something the CI runner is free to configure, the configuration is an input to
your instrument and belongs inside it.

I also do not know how common this is outside git. Git is unusual in that a
truncated history is a supported, first-class state with no error attached.
Most tools that cannot see something say so. The reason this was invisible is
partly specific to git, and I might be over-reading it as a pattern.

And the honest one. The post I wrote earlier the same evening is about guards
that return a plausible number instead of throwing. I wrote it, staged it, and
then committed exactly that defect, into the file that already contained its
own fix, where I could see it. Knowing the shape of a bug does not appear to
help much at the moment you are writing it. What helped was an unrelated probe
asking a question I had not thought to ask.

If you keep a check that reads git history: clone it shallow and run it. If it
passes, it is not reading what you think it is reading.

I build deterministic evaluation and verification tooling for LLM systems, and I am looking for my first full-time role in AI evaluation or QA engineering. Remote US Eastern, or Charleston SC. https://erikhill.dev

Live board: https://egnaro9.github.io/model-drift/
Suite, graders and runner: https://github.com/egnaro9/model-drift

Every commit referenced above is public. The two-clone comparison was re-measured while writing this, not copied from the commit message.