SpecGuard：讓 AI 程式碼審查依循工程標準

Building an evidence-grounded PR reviewer with GitHub, Notion, and Google Gemini.

Most AI code reviewers can inspect a pull request. But can they review it against the engineering standards your team has already agreed on?

That question led me to build SpecGuard.

In real engineering teams, important decisions are scattered across architecture decision records, security policies, API standards, migration guides, and internal documentation. A reviewer can understand what changed in a diff and still miss why that change conflicts with an established rule.

I wanted to explore a better approach: connect code changes with relevant institutional knowledge, then make review results easier to inspect, verify, and challenge.
What I Built
SpecGuard is a full-stack MVP that brings together three sources of work:

• GitHub: Pull request metadata and code diffs
• Notion: Engineering standards and architectural context
• Google Gemini: Structured analysis of code changes against retrieved context The intended workflow is straightforward:
• Submit a GitHub pull request URL.
• Validate the URL and retrieve the PR through the GitHub API.
• Query a configured Notion database and select potentially relevant pages.
• Send the diff and selected context to a server-side model.
• Validate the model response and calculate a transparent triage score.
• Display findings, supporting context, recommendations, and knowledge gaps. The goal is not to make AI the final authority on code quality. It is to give engineers a more contextual starting point for review. The Technology Stack I kept the existing stack instead of rewriting the project:
• React, TypeScript, and Vite for the frontend
• Express and TypeScript for the backend
• Octokit for GitHub API access
• The official Notion SDK for documentation retrieval
• Google Gemini through the server-side @google/genai SDK
• Zod for request and response validation
• Vitest and Supertest for automated tests One important design choice is keeping provider credentials on the server. The browser calls the application API; it should never need direct access to GitHub, Notion, or Gemini secrets. Why Notion Context Matters A generic code reviewer might flag a pattern as suspicious without knowing whether the team has a documented rule governing it.

SpecGuard explores a more contextual workflow: retrieve candidate engineering documents, rank them against the PR content, and provide the selected context to the model.

This retrieval step is deliberately bounded. Sending an entire documentation workspace to a model would increase noise, cost, and the chance of irrelevant context dominating the review.

Keyword relevance is a useful baseline, not a complete retrieval solution. More advanced semantic retrieval, document chunking, and evaluation would be logical next steps.
Structured Output Is Not Enough
Asking a model to return JSON does not automatically make its analysis reliable.

SpecGuard uses a Zod schema to validate the response structure, including finding severity, descriptions, recommendations, evidence fields, and knowledge gaps. The application calculates its triage score from finding severities rather than treating the model's score as a probability of a defect.

That distinction matters: a prioritization heuristic can help organize human attention, but it is not a calibrated measure of software risk.

The next layer of reliability is evidence integrity. A finding should link to a source only when that source actually supports the claim. File paths and line references should be included only when grounded in the diff. Missing evidence should remain visible as uncertainty, not be disguised as confidence.
From Prototype to a Real Workflow
One of the most useful parts of this project was moving beyond a static review dashboard toward a backend-driven flow.

The frontend submits review requests to the Express API and renders the response, with loading and error states. The backend contains the integrations and validation needed to connect GitHub, Notion, and Gemini using configured credentials.

The repository also includes tests for PR URL validation, Notion relevance selection, score calculation, model-response parsing, and basic API behavior. These tests provide a foundation; live service integration and deployment checks remain separate verification steps.

I deliberately kept automatic GitHub review comments, webhooks, Slack notifications, and a full MCP workflow outside the claimed MVP scope. Those features should be added only when their permissions, failure handling, and end-to-end behavior can be verified.
What I Would Improve Next
The next iteration would focus on four areas:

1. Evidence integrity: Verify that every citation supports the specific finding it accompanies.
2. Diff fidelity: Preserve file boundaries, hunk headers, and line information throughout processing.
3. Retrieval quality: Evaluate whether the selected Notion documents are actually relevant to a PR.
4. Integration testing: Exercise the complete flow against controlled GitHub, Notion, and model fixtures, followed by credential-backed integration checks. I would also evaluate the reviewer against a curated set of pull requests with known issues, measuring false positives, missed findings, evidence accuracy, and latency instead of relying on a few impressive demonstrations. What I Learned The interesting engineering problem is not simply connecting an LLM to a GitHub API.

It is building the boundaries around that model: retrieving the right context, validating what comes back, preserving source fidelity, handling missing information, and making the result inspectable by a human.

A useful AI engineering system should not just produce an answer. It should help a user understand where that answer came from, what supports it, and where its limitations begin.

That is the direction I want to take SpecGuard.
Try the Project
GitHub repository: https://github.com/NikhilRaman12/SpecGuard---Notion-MCP-PR-reviewer

SpecGuard is an evolving MVP, not a finished enterprise platform. It requires configured credentials and accessible GitHub and Notion resources to run the integrated workflow. Please check the repository README for setup instructions and current limitations.

I would love to hear your thoughts: what would you prioritize first in an evidence-grounded code reviewer—retrieval quality, citation verification, or evaluation against known defects?

mlhacks #ai #typescript #github #notion #opensource