OpenAI、Claude、Gemini 文字摘要 API 品質與成本比較：按租戶制定程式碼審查規則

TL;DR: For a marketplace that turns code changes into structured review findings, the useful comparison is not a leaderboard. It is a repeatable test that records summary quality, schema validity, and spend per tenant. OpenAI, Claude, and Gemini can all sit behind the same application contract, but the contract must own the evaluation. That keeps a weak summary from looking cheap and a verbose summary from looking smart.

I run a one-person SaaS. My constraint is revenue per hour, not theoretical model preference. I ship weekly, and I outsource the undifferentiated parts when the boundary is clear. A small evaluation harness is worth building because it keeps provider switching, prompt edits, and tenant pricing from becoming one tangled guess.

The decision matrix I would use

The recommendation is boring: keep the application contract stable and test each API against the same labeled changes. Select a configuration only after it meets the quality floor and exposes enough usage data to explain a tenant's bill. The matrix is a measurement plan, not a ranking.

What does “quality” mean for a code-review summary?

A summary can be fluent and still fail the job. In a marketplace, a finding that omits the affected seller flow is hard to act on. A finding that invents a severity wastes an engineer's limited review time. A response that drops one required field breaks downstream routing.

I would score four separate things:

1. Finding precision: does a reported issue point to a real risk in the change?
2. Omission rate: did the response miss a labeled issue?
3. Contract validity: does every finding have the required location, severity, explanation, and suggested next step?
4. Summary usefulness: can an engineer understand the change boundary without rereading the whole diff?

Do not collapse those into one attractive number. A provider can improve prose while losing precision. Your evaluator should retain the raw response, the parsed findings, and the human label. Start with 50 representative changes if you need a manageable pilot; include small fixes, permission changes, seller-facing checkout work, and deliberately noisy diffs. The point is coverage of failure modes, not a ceremonial benchmark.

The prompt should ask for a bounded result. The surrounding service should enforce it. A generic TypeScript boundary is enough:

type Finding = {
  path: string;
  line: number | null;
  severity: "low" | "medium" | "high" | "critical";
  summary: string;
  nextStep: string;
};

type ReviewResult = {
  tenantId: string;
  changeId: string;
  findings: Finding[];
  inputTokens: number;
  outputTokens: number;
  provider: string;
};

function validateResult(result: ReviewResult): void {
  if (!result.tenantId || !result.changeId) throw new Error("missing identity");
  for (const finding of result.findings) {
    if (!finding.path || !finding.summary || !finding.nextStep) {
      throw new Error("invalid finding");
    }
  }
}

The union above is an application label, not a claim that the providers share one native schema. Adapters translate their responses into it, then validation and accounting happen in one place. That is where a solo founder gets leverage. The boring part becomes reusable.

How should per-tenant cost be measured?

Cost visibility starts before an API call. Attach a tenant id, change id, and evaluation version to every request. Store input and output usage returned by the API, the provider label, the selected model label, and whether the result was retried. If a request fails before usage is returned, record that fact as an unknown rather than silently writing zero.

A simple ledger row can look like this:

type UsageRow = {
  tenantId: string;
  changeId: string;
  provider: string;
  model: string;
  inputTokens: number | null;
  outputTokens: number | null;
  retryCount: number;
  evaluationVersion: string;
};

The calculation should be explicit and versioned. Pin the evaluation rules to a small integer, such as version 7, so a prompt change cannot masquerade as a provider change. Keep unit rates outside the review result so a later rate change does not rewrite historical usage. More importantly, report cost beside quality: cost per accepted finding, cost per reviewed change, and the share of requests that needed a retry. A low raw request cost is not useful if the output needs a second pass or an engineer has to discard it. The trade-off is plain: an inexpensive request with unusable findings has negative revenue per hour.

That is the trap.

For token accounting, use the provider's usage fields as the billing record. A tokenizer library can help estimate input size before a request, but an estimate is not a bill. The public tiktoken library is useful evidence for how one provider exposes tokenization; it is not a universal meter for all three APIs. That distinction prevents a very common spreadsheet error.

What makes an API text summarization comparison fair?

OpenAI, Claude, and Gemini are three real integration targets, not interchangeable strings in an environment variable. They differ in request formats, response details, usage reporting, model identifiers, and the mechanics available for constrained output. Those differences belong in adapters and tests. They should not leak into tenant-facing billing logic.

The fair comparison is a compatibility table generated by your own harness. For each target, capture whether the same prompt can produce parseable findings, whether usage is present, how retries behave, and how human reviewers score the result. Do this on identical change sets. Do not infer summary quality from model names or from one impressive example.

A reranker may be useful later if the review workflow retrieves relevant files or prior findings, but that is a separate operation from summarization. Rerank output changes which evidence reaches the summarizer; it does not make the final summary correct by itself. Keeping those stages separate lets you attribute tokens and quality to the right tenant operation.

The runner-up is better when the primary requirement is different. If a team already has a strong adapter, evaluator, and observability path for another API, migration cost may outweigh a small quality difference. If the workload has long diffs, the decisive test may be context handling and truncation behavior. If findings must follow a strict schema, parse failures and repair calls may matter more than prose preference. The right answer can change as the marketplace changes.

The weekly operating rule

I would make the harness part of the weekly shipping loop. Every prompt or adapter change gets a fixed sample, a schema-validity check, a human review of disagreements, and a tenant-level usage report. Keep the sample stable long enough to see regressions. Add new labeled changes when a production pattern exposes a blind spot.

This is also where outsourcing helps. A hosted model can handle language generation; your service should own identity, validation, accounting, and audit history. Those are product responsibilities. Handing them to a prompt is how cost visibility disappears.

The practical decision rule is short: reject any configuration that misses the quality floor, cannot explain usage by tenant, or requires unbounded repair calls. Among the remaining options, choose the one that gives the team the most reliable accepted finding per unit of engineering attention. That is a decision a marketplace can revisit without rewriting its review product.

Sources

• OpenAI, tiktoken tokenizer library: https://github.com/openai/tiktoken
• Cohere, Rerank overview: https://docs.cohere.com/docs/rerank-overview
• OpenAI API documentation: https://platform.openai.com/docs/api-reference
• Anthropic API documentation: https://docs.anthropic.com/en/api
• Google Gemini API documentation: https://ai.google.dev/api