10 項檢查防止未經驗證的 AI 生成程式碼流入正式環境

Bookmark It! STOP Shipping Broken Vibe Code: 10 Checks That Save Your App at 2 AM

You know the scene: it's 2 AM, your AI agent just deployed a "minor fix," and now the entire app is returning 500s. The code looked right when the model generated it. The tests passed locally. But in production, it's a disaster. This is the cost of shipping vibe code—code that feels right but hasn't been verified.

AI coding agents are powerful, but they are also confidently wrong. They hallucinate APIs, ignore edge cases, and happily introduce security holes. If you're building agents that write or modify code, you need a guardrail layer. Below are 10 checks you can implement in Python to catch broken vibe code before it hits production. Each check is runnable and designed to be integrated into your agent's output pipeline.

The Problem: Agents Ship Vibes, Not Verified Code

An AI agent loop typically does: plan → generate code → execute → observe → repeat. Without explicit termination conditions, the loop can run forever. Without verification, it can "succeed" by producing code that runs but is semantically wrong. Worse, if your agent executes arbitrary code (e.g., via exec or subprocess), you're one prompt injection away from a remote code execution vulnerability.

The solution is a check pipeline: a set of deterministic validations applied to every code artifact before it is accepted. These checks are not a replacement for human review, but they catch the low-hanging fruit that causes 2 AM pages.

The 10 Checks

We'll implement each as a function that takes a code string (and optional context) and returns a boolean or raises an exception. We'll then compose them into a pipeline.

1. Syntax Validation

Use Python's ast.parse to ensure the code is syntactically valid.

import ast

def check_syntax(code: str) -> bool:
    try:
        ast.parse(code)
        return True
    except SyntaxError as e:
        print(f"Syntax error: {e}")
        return False

2. Import Allowlist

Agents love to import random packages. Restrict imports to a known allowlist.

def check_imports(code: str, allowed: set[str]) -> bool:
    tree = ast.parse(code)
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            for alias in node.names:
                if alias.name.split('.')[0] not in allowed:
                    print(f"Disallowed import: {alias.name}")
                    return False
        elif isinstance(node, ast.ImportFrom):
            if node.module and node.module.split('.')[0] not in allowed:
                print(f"Disallowed import: {node.module}")
                return False
    return True

3. No Dangerous Builtins

Block eval, exec, compile, and __import__ unless explicitly allowed. Security warning: these functions can execute arbitrary code. Never enable them in untrusted contexts.

DANGEROUS = {'eval', 'exec', 'compile', '__import__'}

def check_dangerous_builtins(code: str) -> bool:
    tree = ast.parse(code)
    for node in ast.walk(tree):
        if isinstance(node, ast.Call):
            if isinstance(node.func, ast.Name) and node.func.id in DANGEROUS:
                print(f"Dangerous call: {node.func.id}")
                return False
    return True

4. No Subprocess or OS System Calls

Block subprocess, os.system, and similar modules unless the agent is explicitly authorized.

def check_subprocess(code: str) -> bool:
    tree = ast.parse(code)
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            for alias in node.names:
                if alias.name in {'subprocess', 'os'}:
                    print(f"Disallowed module: {alias.name}")
                    return False
        elif isinstance(node, ast.ImportFrom):
            if node.module in {'subprocess', 'os'}:
                print(f"Disallowed module: {node.module}")
                return False
    return True

5. Function Signature Check

If the agent is modifying an existing function, ensure the signature matches the original. This prevents accidental API breaks.

def check_function_signature(code: str, func_name: str, expected_params: list[str]) -> bool:
    tree = ast.parse(code)
    for node in ast.walk(tree):
        if isinstance(node, ast.FunctionDef) and node.name == func_name:
            params = [arg.arg for arg in node.args.args]
            if params != expected_params:
                print(f"Signature mismatch for {func_name}: {params} != {expected_params}")
                return False
            return True
    print(f"Function {func_name} not found")
    return False

6. Test Coverage Threshold

Run the agent's generated tests and require a minimum coverage. Use coverage.py programmatically.

import coverage

def check_coverage(code: str, min_percent: float = 80.0) -> bool:
    # Write code to a temp file and run coverage
    # Simplified for brevity; in practice, use a temp directory
    import tempfile, os
    with tempfile.NamedTemporaryFile('w', suffix='.py', delete=False) as f:
        f.write(code)
        path = f.name
    cov = coverage.Coverage(source=[path])
    cov.start()
    # Execute the code (in a sandbox!)
    try:
        exec(compile(code, path, 'exec'), {})
    except Exception as e:
        print(f"Execution error: {e}")
        return False
    finally:
        cov.stop()
        cov.save()
    percent = cov.report()
    os.unlink(path)
    return percent >= min_percent

Security warning: exec here runs arbitrary code. Only use this in an isolated sandbox (e.g., Docker container with no network). Never run untrusted code on your host.

7. Lint with Flake8

Catch style issues and potential bugs.

import subprocess

def check_lint(code: str) -> bool:
    import tempfile, os
    with tempfile.NamedTemporaryFile('w', suffix='.py', delete=False) as f:
        f.write(code)
        path = f.name
    result = subprocess.run(['flake8', path], capture_output=True, text=True)
    os.unlink(path)
    if result.returncode != 0:
        print(result.stdout)
        return False
    return True

8. Type Check with Mypy

Static type checking catches many runtime errors.

def check_types(code: str) -> bool:
    import tempfile, os
    with tempfile.NamedTemporaryFile('w', suffix='.py', delete=False) as f:
        f.write(code)
        path = f.name
    result = subprocess.run(['mypy', path], capture_output=True, text=True)
    os.unlink(path)
    if result.returncode != 0:
        print(result.stdout)
        return False
    return True

9. Dependency Audit

If the agent adds dependencies, check them against a vulnerability database (e.g., pip-audit).

def check_dependencies(requirements: str) -> bool:
    import tempfile, os
    with tempfile.NamedTemporaryFile('w', suffix='.txt', delete=False) as f:
        f.write(requirements)
        path = f.name
    result = subprocess.run(['pip-audit', '-r', path], capture_output=True, text=True)
    os.unlink(path)
    if result.returncode != 0:
        print(result.stdout)
        return False
    return True

10. Explicit Agent Loop Termination

If your agent uses a loop, define a maximum iteration count and a success condition. Without this, the agent may never stop.

MAX_ITERATIONS = 10

def agent_loop(agent, task, success_check):
    for i in range(MAX_ITERATIONS):
        code = agent.generate(task)
        if success_check(code):
            return code
        task = agent.refine(task, code)
    raise RuntimeError("Agent loop exceeded maximum iterations without success")

Implementation: The Check Pipeline

Compose the checks into a single function. Use a list of (name, check) tuples. Each check returns True if passed.

def validate_code(code: str, allowed_imports: set[str], expected_func: str | None = None) -> bool:
    checks = [
        ("syntax", lambda c: check_syntax(c)),
        ("imports", lambda c: check_imports(c, allowed_imports)),
        ("dangerous", lambda c: check_dangerous_builtins(c)),
        ("subprocess", lambda c: check_subprocess(c)),
        ("lint", lambda c: check_lint(c)),
        ("types", lambda c: check_types(c)),
    ]
    if expected_func:
        checks.append(("signature", lambda c: check_function_signature(c, expected_func, ['x'])))
    for name, check in checks:
        if not check(code):
            print(f"Check failed: {name}")
            return False
    return True

Integrate this into your agent's output step: only accept code that passes all checks. For execution, always use a sandbox (e.g., docker run --rm -v $(pwd):/code python:3.11-slim).

Key Takeaways

• Vibe code is a liability. Verify every artifact with deterministic checks.
• Security first. Never exec untrusted code on your host. Use containers.
• Explicit termination. Agent loops must have a max iteration count and a clear success condition.
• Compose checks. A pipeline of small, focused validators is easier to maintain than one monolithic validator.
• Bookmark this. At 2 AM, you'll want these checks in your back pocket.

These 10 checks won't catch every bug, but they'll stop the most common causes of broken deployments. Add them to your agent's CI/CD, and sleep better.