如何評估模型撰寫文件章節的穩定性

A doc section a model can write five different ways is not a section the model owns; it is a question your source material has not answered yet. Most documentation-generation workflows judge a draft by reading it once, which rewards fluency and hides disagreement. The workflow below replaces that judgement with a measurement: write the same section prompt several times, score how much the wording moves, and let the score decide who writes the final text. The artifact is a small Python harness you can run offline against a fixture before you point it at any live endpoint.

The problem with a single sample

One model draft reads like a decision because it is coherent, specific, and finished. Run the same prompt again and you often get a different number, a different order, a different claim about what happens on failure. That variance is information about your inputs, not noise to average away. A section that changes shape between samples usually means the underlying spec, config, or error text is ambiguous or absent.

So the useful question is not "is this draft good?" but "does this draft converge?". Convergence is cheap to measure, hard to fake, and directly actionable for ownership.

The rule: k samples, two signals, three verdicts

Collect at least three samples per section, five if you can afford it. Compute a mean pairwise similarity over normalized token shingles, then separately detect claim-like tokens (numbers, sizes, durations, URLs) that appear in some samples but not all. The similarity tells you whether the wording is settled; the divergent-claim check tells you whether the facts are.

Divergent claims override the score deliberately, because a section that confidently cites three retry counts is worse than a section that cites none.

Step 1: The harness

Save this as draft_stability.py. It has no network code, so the same file works in CI, on a laptop, and on a scheduled job.

#!/usr/bin/env python3
"""draft_stability.py — score how settled a doc section is across repeated model runs.

Fixture mode (no keys, no network):
    python draft_stability.py --fixture samples.json --k 3 --out record.md
"""
from __future__ import annotations

import argparse
import itertools
import json
import re
from pathlib import Path

WORD = re.compile(r"[A-Za-z_][A-Za-z0-9_]*|\d+(?:[.,]\d+)*|\S")
CLAIMY = re.compile(r"\b\d+(?:[.,]\d+)*(?:\s?(?:ms|s|min|h|%|MB|GB))?\b|https?://\S+")

K = 5  # shingle width, in tokens


def tokens(text: str) -> list[str]:
    return WORD.findall(text)


def shingles(text: str, k: int = K) -> set[tuple[str, ...]]:
    t = [w.lower() for w in tokens(text)]
    if not t:
        return set()
    if len(t) < k:
        return {tuple(t)}
    return {tuple(t[i : i + k]) for i in range(len(t) - k + 1)}


def jaccard(a: set, b: set) -> float:
    if not a and not b:
        return 1.0
    return len(a & b) / len(a | b)


def stability(samples: list[str]) -> float:
    pairs = list(itertools.combinations(samples, 2))
    if not pairs:
        return 1.0
    return sum(jaccard(shingles(a), shingles(b)) for a, b in pairs) / len(pairs)


def divergent_claims(samples: list[str]) -> list[str]:
    """Claim-like tokens present in some samples but not in all of them."""
    counts: dict[str, int] = {}
    for text in samples:
        for tok in set(CLAIMY.findall(text)):
            counts[tok] = counts.get(tok, 0) + 1
    n = len(samples)
    return sorted(tok for tok, c in counts.items() if 0 < c < n)


def verdict(score: float, divergent: list[str]) -> tuple[str, str]:
    if divergent:
        return "HUMAN", "claims disagree across samples: " + ", ".join(divergent[:5])
    if score >= 0.90:
        return "MODEL-DRAFT / HUMAN-APPROVE", "wording is stable; review tone and correctness"
    if score >= 0.75:
        return "MODEL-OUTLINE / HUMAN-NORMATIVE", "model proposes structure; human writes binding sentences"
    return "HUMAN", "wording is unsettled; the input material is probably ambiguous"


def render(results: dict, k: int) -> str:
    lines = ["<!-- draft-stability record", f"samples_per_section: {k}"]
    for name, r in results.items():
        lines.append(f"{name}: stability={r['stability']:.2f} verdict={r['verdict']}")
    lines.append("-->")
    return "\n".join(lines)


def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--fixture", required=True, type=Path)
    ap.add_argument("--k", type=int, default=3)
    ap.add_argument("--out", type=Path)
    args = ap.parse_args()

    data = json.loads(args.fixture.read_text(encoding="utf-8"))
    results = {}
    for name, samples in data["sections"].items():
        if len(samples) < args.k:
            raise SystemExit(f"{name}: needs >= {args.k} samples, found {len(samples)}")
        use = samples[: args.k]
        score = stability(use)
        div = divergent_claims(use)
        label, why = verdict(score, div)
        results[name] = {"stability": round(score, 3), "verdict": label, "why": why}

    width = max(len(n) for n in results)
    for name, r in results.items():
        print(f"{name:<{width}}  {r['stability']:.2f}  {r['verdict']}")
        print(f"{'':<{width}}        {r['why']}")

    if args.out:
        args.out.write_text(render(results, args.k) + "\n", encoding="utf-8")


if __name__ == "__main__":
    main()

Step 2: A fixture that proves the harness, not the model

Create samples.json with canned samples. The point is to verify your scoring path end to end before any live call, and to keep the check running when an endpoint is unavailable.

{
  "sections": {
    "install": [
      "Run pip install example-sdk in a clean virtual environment, then confirm the installed version with example-sdk --version. If the command is missing from your shell path, activate the environment again and retry the install.",
      "Run pip install example-sdk in a clean virtual environment and confirm the installed version with example-sdk --version. If the command is missing from your shell path, activate the environment again before retrying the install.",
      "Run pip install example-sdk in a clean virtual environment, then confirm the installed version with example-sdk --version. If the command is missing from your path, activate the environment again and retry the install."
    ],
    "retry-behavior": [
      "The client retries 3 times with exponential backoff starting at 200 ms and gives up after 5 s. A retry is attempted only when the response is 503 or 504.",
      "The client retries 5 times with exponential backoff starting at 100 ms and gives up after 30 s. A retry is attempted on any 5xx response, including 500.",
      "The client retries 2 times with a fixed 250 ms delay and no total timeout. Retries apply to 503 responses only and never to 500."
    ]
  }
}

Run python draft_stability.py --fixture samples.json --k 3 --out record.md. The fixture is tuned so install converges and retry-behavior fails the divergence check; the exact scores describe these three strings and nothing else. Expect roughly:

install         0.9x  MODEL-DRAFT / HUMAN-APPROVE
retry-behavior  0.xx  HUMAN
                      claims disagree across samples: 100, 2, 200, 250, 3, 30, 5, 503, 504, 500

That second line is the whole method in one output: the model can write retry behavior fluently, but the retry policy has not been decided anywhere a model can read it.

Step 3: Collect live samples with a one-function contract

The harness deliberately knows nothing about your client. Wire your own, keeping the contract to one function.

# Wire whichever client you already use; the harness only needs this shape.
def backend(prompt: str) -> str:
    ...

PROMPT = "Write the 'retry behavior' section for the example-sdk docs. Do not invent limits."
samples = {"sections": {"retry-behavior": [backend(PROMPT) for _ in range(5)]}}
Path("live.json").write_text(json.dumps(samples, indent=2), encoding="utf-8")

Then run python draft_stability.py --fixture live.json --k 5 --out record.md. The --k 5 recomputes over all five samples and the record file records the inputs you actually used.

Step 4: Turn the verdict into an assignment

MODEL-DRAFT means an editor reads for tone and a reviewer checks the commands; MODEL-OUTLINE means a human must write every sentence that states a limit, guarantee, or default. HUMAN means the next task is not writing at all: find the config file, incident, or owner that settles the question, and only then return to the prompt.

Step 5: Re-run when the inputs move

Stability is a property of a moment, not a permanent label. Re-run the harness when the config, error strings, or release notes that feed a section change, and treat a falling score as a signal that the section's source of truth drifted. Commit record.md next to the page so reviewers see which sections were measured and which were simply written.

Where MonkeyCode fits, and what it does not prove

Repeated sampling costs calls, which is the practical argument for a free-model path: the method needs several generations of the same prompt, not one expensive one. MonkeyCode's free model access covers that loop with the harness unchanged, and the free server option gives the fixture-mode job a place to run on a schedule so the record stays current. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Availability is operator-supplied and not a durability guarantee, so keep the fixture run in CI as the floor: the check must still execute and still fail correctly when the live endpoint is unreachable. If you only ever run fixture mode, you have tested the scorer, not your documentation.

Limitations, and who should skip this

Similarity measures wording, not truth: five samples can agree on a wrong retry count and score beautifully, which is why the divergent-claim check exists and why a human still reviews MODEL-DRAFT. A deterministic sampler inflates every score, so sample with the variation you actually ship. Very short sections degrade into a single shingle, changelogs and generated examples are unstable by design, and legal or contractual text should never be assigned by a score. Anyone looking for a benchmark of model quality should not use this; it measures whether your inputs are settled.

If you want to try the sampling loop on free model access, the harness above runs unchanged, fixture first.