FlakePilot 以確定性判定分析不穩定測試，Gemma 只負責提出假設

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

A test fails in CI. You press "re-run". It passes. You merge, and a week later it fails again on someone else's pull request.

I have done this far more often than I'd like to admit. Re-running is the lazy move, and it's lazy in the bad way: it throws away the evidence. The first failure, the environment, the commit and the attempt that followed it are exactly what you'd need to decide whether the test is flaky or whether you just broke something. FlakePilot is my attempt at being lazy in the good way: file the evidence once, let a fixed set of rules read it, and stop guessing.

What I Built

FlakePilot is an open-source (Apache-2.0), read-only control plane for investigating flaky tests.

CI uploads a JUnit XML report. FlakePilot stores it with who, where and when: run id, attempt, job, commit SHA, ref, environment.
It rebuilds each test's attempt chain (failed → passed, failed → failed, and so on).

It classifies every failure with a versioned, deterministic rule set, ti-classification/1. The result is flaky, deterministic_regression, or an honest inconclusive such as insufficient_history. There are no confidence scores, because the data doesn't support them.
Every verdict is stored together with the evidence it was computed from, so you can see why.
On request, an investigation can draft a hypothesis about the cause with an open-weight model (Gemma). Code then checks that every claim cites stored evidence.

The effort it saves is the rerun-and-wonder loop. The user is whoever triages flaky CI at a small team or in an open-source project: the person who is currently reading three logs to decide whether to press "re-run".

I want to be upfront about the weight of the AI here, because the challenge asks for open-source AI at the core. In FlakePilot the verdict is never a model's output. Gemma is an optional, off-by-default advisory stage. I'll explain below why I think that's the right design, and where it leaves this submission.

Demo

Deployed control plane: app.flakepilot.tech. It uses GitHub sign-in, so the signed-in pages need a login. The public landing page is open.
I have no screen recording to show, so I'm not going to pretend there is one. What you can open without logging in is the landing page and the repository. The signed-in run view is what the diagrams below describe.

Please note: the execution plane is closed. FlakePilot never runs repository code, ever. It only reads reports.

Code

CodewithJha / FlakePilot

Evidence-backed flaky-test investigation: ingest CI results, classify failures, propose and verify minimal patches with human approval.

The repository is public and Apache-2.0. The first commit is from 20 October 2026 (19:32 IST), inside the challenge window, and the history is in the repository. If any commits land after the deadline, they are noted in the README.

How I Built It

The shape of the system

The stack is Python 3.12, FastAPI and PostgreSQL (SQLAlchemy, Alembic, asyncpg), managed with uv, and checked with Ruff, strict mypy, import-linter and pytest. It is a modular monolith with bounded contexts, and the import boundaries between them are enforced by 21 import-linter contracts in CI. The UI is a small React + TypeScript app built with Vite. Python still does every authorization check and only then embeds the run's data in the page.

Fail-closed on purpose

Most of the decisions were about what the system refuses to do:

Missing configuration disables a feature instead of guessing a default.
A resource another tenant can see is always 404 not_found, never 403.
A commit SHA is stored exactly as CI reported it. Its revision status stays unchecked, because FlakePilot does not verify Git ancestry.
A test with too little history is inconclusive, not a coin flip.
Nothing executes repository code.

Where Gemma fits

The investigation walks bounded stages: collect, research, browse, hypothesize, prioritize, brief. Each stage is skipped with a stated reason when its provider isn't configured. The hypothesize stage asks a model to suggest candidate causes from the stored evidence. It is off unless an organisation admin turns on an AI policy, and its output is stored as advisory data. It never changes a verdict and never grants anything.

The part I'm happiest with is the checking step. The model's answer has to parse as a fixed JSON shape, and each claim has to cite an evidence id that actually exists for that run. A claim that cites nothing, or cites something that isn't there, is rejected.

Why Does Open Innovation Matter?

Honestly, for this project the reason is trust and swappability, not capability.

The verdict doesn't depend on a model at all. That's deliberate. If the model is wrong, slow, or retired, the classification is the same. An open-weight model fits an advisory role: I can change it without touching anything that decides a verdict.
The model is a configuration choice. The hypothesize stage talks to either an operator-provided local endpoint or hosted Gemma. An operator who doesn't want failure logs leaving their network can point it at their own server.
Open code means the rules can be audited. ti-classification/1 is a plain, versioned rule set in a public repository. If you disagree with a verdict, you can read exactly why it was produced.

What I verified, and what I didn't:

Verified live (24–25 Oct 2026): hosted Gemma 4 (gemma-4-31b-it) through the Gemini API free tier. One investigation of a seeded flaky run, on a throwaway PostgreSQL database, produced an evidence-linked hypothesis with zero unverified claims.
Not verified: I haven't recorded a run against a local Gemma endpoint, so the "your logs never leave the machine" option is supported by the code and unit tests but not by a live run. Also, the hosted free tier may use prompts, which is a real trade-off.

So: open-weight AI is a deliberate, replaceable part of this project, but it is not the thing that makes the project work. If the challenge reads "open-source AI at the core" strictly, this is a borderline entry and I'd rather say so than dress it up.

What did not work

Gemma's first answers all failed validation. My prompt never said that cause had to be a snake_case code or that citations had to be {"evidence_id": ...} objects, so real answers came back as invalid_hypothesis. The fix was to spell out every key and end the prompt with an example that the validator accepts. A unit test checks that example against the validator so the two can't drift apart. (Commit fd71d89.)
A hosted call can simply fail. In one earlier end-to-end run the hypothesize stage was skipped with provider_http_error after the bounded retries. The system recorded that honestly and carried on, which is the behaviour I wanted, but it is also a reminder that a hosted free tier is not a dependency you want in the critical path.
Several sponsor integrations are not in the product. Some were evaluated and deliberately not adopted, or exercised only against synthetic data or a local dev server. I'm not claiming them here.

Current limitations

Pre-alpha. The README says "not production-ready", and I mean it.
The execution plane is closed, so FlakePilot can't reproduce a failure itself. It only reads what CI already recorded.
Raw report bodies are stored as plaintext files. There is no encryption at rest yet.
The signed-in demo needs GitHub sign-in, so a quick look from outside is limited to the landing page and the code.
The classifier needs history. With one run, most things are honestly inconclusive.

The Lazy Payoff

I haven't measured time saved, and I'm not going to invent a number. The qualitative payoff is concrete, though: instead of pressing "re-run" and hoping, you get a stored attempt chain and a verdict that names the evidence behind it. If you want me to measure it, the honest experiment is to count how often a triage question gets answered without opening a raw log. I haven't run that yet.

What I learned

A model that proposes while code verifies is easier to trust than a model that decides. Rejecting an unsupported claim is cheap and visible.
"Inconclusive" is a feature. Making the system say it out loud forced me to design for missing history instead of hiding it.
Writing the prompt's expected output down, with a validated example, solved more than any prompt tweaking did.

Next I'd like to run the hypothesize stage against a local Gemma endpoint end to end and record it, add a no-login demo, and measure the triage claim above.

Prize Categories

Best Use of Gemma: the hypothesize stage uses Gemma 4 served through the Gemini API (live-verified as described above).
.tech domain: the deployment runs on app.flakepilot.tech under a flakepilot.tech domain.

(I'm deliberately not claiming other partner categories. Some of them appear in the repository, but I'd rather only declare what I can show.)