Agent.reviews 如何讓 AI 智能體撰寫產品評價，並揭示工具回饋循環的問題

Armature (YC P26) measured 50,000+ agent sessions and found agents repeatedly hit the same tool limitations. The feedback loop that would fix these issues doesn't exist. Human users file GitHub issues or support tickets. Agents just fail silently, retry with exponential backoff, or hallucinate workarounds.

Agent.reviews is Armature's response: a platform where agents write reviews based on actual usage patterns. The site indexes tools (FastAPI, Supabase, Auth0, Modal, uv, OpenAI API) and publishes agent-generated reviews with numeric ratings. Each review describes what the agent tried, what worked, and what broke.

The interesting part is not the review content. It's the instrumentation layer that captures agent behavior at scale, the schema that differentiates agent feedback from human feedback, and the security boundary that prevents vendors from gaming the system.

The Instrumentation Layer

To generate reviews from 50k+ sessions, you need telemetry that captures:

• Tool invocation traces: which API endpoint, which parameters, which response codes
• Retry patterns: how many attempts, which backoff strategy, which error messages triggered abandonment
• Context switches: when the agent pivoted to a different tool or asked for human intervention
• Success metrics: task completion rate, time to first success, number of fallback strategies

Armature's approach instruments coding agents specifically. The platform provides a skill definition (https://agent.reviews/skill.md) and setup steps (https://agent.reviews/install.md) that agents read to understand how to submit reviews. Every page has a Markdown version (address plus .md) listed in https://agent.reviews/llms.txt.

This is a crawlable, agent-readable schema. The agent doesn't need a custom SDK. It reads the instructions, follows the protocol, and submits structured feedback after completing a task.

Agent Review Schema vs. Human Review Schema

Human reviews rate usability, documentation quality, and customer support responsiveness. Agent reviews rate different dimensions:

Agent.reviews publishes numeric ratings (FastAPI: 4.6, Supabase: 4.3, Auth0: 4.0) but the real signal is in the prose. For example:

"New parts since the earlier review today: a lifespan context manager ran database migrations on startup, BackgroundTasks queued the slow scoring work after a 202 response, Header(default=...) read a shared-secret header, and a Literal-typed dropdown field rejected bad values."

This is not a human describing their experience. This is an agent describing which code paths it executed, which patterns it observed, and which guardrails it encountered.

Security Boundary: Preventing Review Gaming

The obvious attack vector: tool vendors prompt-inject positive reviews or submit fake execution traces. Agent.reviews mitigates this by requiring verified execution traces.

The platform states: "Never your code, prompts or secrets. Your name is never shown." This implies:

1. Trace verification: reviews must include cryptographically signed execution logs or API call receipts
2. Anonymization: the agent's identity (and by extension, the vendor's identity) is stripped before publication
3. Replay detection: duplicate traces from the same session are deduplicated

The security model resembles certificate transparency logs. You can't fake a review without faking the underlying API interaction, and faking the API interaction requires compromising the tool itself.

One unresolved question: can a vendor run their own agent, instrument it to always succeed, and flood the platform with positive reviews? The answer depends on whether Armature cross-references reviews with independent session telemetry. If they only accept reviews from agents they instrument, the attack surface shrinks. If they accept reviews from any agent that follows the protocol, they need reputation scoring or stake-based submission.

Feedback Loop Latency

The value proposition is speed: how fast can a tool maintainer see agent failures and ship a fix?

Traditional feedback loops:

1. User hits a bug
2. User files a GitHub issue (if they bother)
3. Maintainer triages the issue (days to weeks)
4. Maintainer ships a fix (weeks to months)
5. User upgrades and retries (months to never)

Agent feedback loops:

1. Agent hits a bug
2. Agent writes a review immediately
3. Review appears on agent.reviews (minutes)
4. Maintainer sees the review (hours to days)
5. Maintainer ships a fix (days to weeks)
6. Next agent session tests the fix automatically (minutes after deploy)

The latency improvement is in steps 1-3. Agents don't procrastinate. They don't forget to file issues. They write reviews as part of their task completion protocol.

The remaining bottleneck is step 4: maintainer notification. If agent.reviews pushes reviews to a webhook or GitHub issue automatically, the loop tightens further. If maintainers have to poll the site, adoption suffers.

Architecture: Crawlable Agent Protocol

Agent.reviews uses a crawlable protocol instead of a REST API. This is a deliberate design choice.

Why crawlable beats REST for agents:

• Agents already know how to fetch and parse Markdown
• No SDK versioning or dependency hell
• No authentication flow (reviews are public)
• No rate limiting on reads (only writes need throttling)
• Easier to cache and mirror

The protocol:

1. Agent reads https://agent.reviews/llms.txt to discover available endpoints
2. Agent fetches tool pages (e.g., https://agent.reviews/fastapi.md)
3. Agent reads existing reviews to understand rating criteria
4. Agent submits a new review by POSTing structured Markdown to a submission endpoint

This is similar to how RSS feeds work, but optimized for LLM consumption. The .md suffix convention signals machine-readable content without requiring content negotiation headers.

Observability: What Gets Measured

Agent.reviews exposes:

• Review count per tool: FastAPI has 1,838 reviews, Supabase has 1,214
• Rating distribution: numeric scores (4.0 to 4.7 in the sample)
• Recency signals: "New parts since the earlier review today" indicates continuous monitoring
• Agent diversity: reviews mention Claude Code, Codex, and Muse Code as distinct agent identities

The platform does not expose:

• Individual agent session IDs (privacy boundary)
• Vendor response times (no SLA tracking yet)
• Comparative benchmarks (no head-to-head tool rankings)

The missing piece is trend data. If a tool's rating drops from 4.6 to 4.2 over a week, maintainers need to know which change caused the regression. Agent.reviews could publish time-series charts or diff views between rating periods.

Deployment Shape

Agent.reviews is a public web service with no authentication required for reads. This implies:

• Static site generation: tool pages are pre-rendered Markdown, not dynamic queries
• Write-path separation: review submissions go through a separate ingestion pipeline
• CDN distribution: .md files are cacheable and globally distributed
• Async processing: review validation and scoring happen out-of-band

The write path likely includes:

# Pseudocode for review submission
def submit_review(agent_id, tool_name, trace_data, rating, prose):
    # Verify execution trace signature
    if not verify_trace(trace_data):
        return {"error": "Invalid execution trace"}

    # Anonymize agent identity
    anon_id = hash_agent_id(agent_id)

    # Check for duplicate submissions
    if is_duplicate(anon_id, tool_name, trace_data):
        return {"error": "Duplicate review"}

    # Queue for moderation and publication
    queue_review({
        "tool": tool_name,
        "rating": rating,
        "prose": prose,
        "trace_hash": hash(trace_data),
        "submitted_at": now()
    })

    return {"status": "queued"}

The moderation step is critical. Without it, the platform becomes a spam vector. With it, the feedback loop latency increases.

Likely Failure Modes

1. Review spam from adversarial agents

If submission is open to any agent, expect:

• Competitors submitting negative reviews with fake traces
• Vendors submitting positive reviews with synthetic workloads
• Pranksters submitting absurd reviews to test moderation

Mitigation: require agents to register with Armature's instrumentation layer before submitting reviews. This adds friction but increases trust.

2. Trace verification bypass

If trace signatures are weak or optional, vendors can forge positive reviews. If trace verification is too strict, legitimate agents can't submit reviews due to clock skew, network issues, or API changes.

Mitigation: use a grace period for trace freshness (e.g., accept traces up to 24 hours old) and allow agents to resubmit with updated traces.

3. Rating inflation over time

As tools improve, ratings should increase. But if ratings only go up, the signal degrades. A 4.6 rating means nothing if every tool eventually reaches 4.8.

Mitigation: normalize ratings within categories or time windows. Show relative rank instead of absolute score.

4. Vendor non-response

If maintainers ignore agent reviews, the feedback loop breaks. Agents keep hitting the same bugs, reviews pile up, and the platform becomes a graveyard of unresolved issues.

Mitigation: surface review counts and response rates on vendor profiles. Gamify maintainer engagement.

Technical Verdict

Use agent.reviews when:

• You maintain a developer tool and want to understand how agents interact with your API
• You're building an agent and need to choose between competing tools based on agent-specific criteria
• You want to instrument agent sessions and need a reference schema for feedback capture

Avoid agent.reviews when:

• You need real-time feedback (the moderation queue adds latency)
• You're optimizing for human users (agent reviews don't correlate with human satisfaction)
• You require private feedback channels (all reviews are public)

The platform's real contribution is not the review content. It's the proof that agent-generated feedback can be structured, verified, and published at scale. The next step is closing the loop: automated issue creation, fix verification, and continuous rating updates.

Source Links

• Agent.reviews
• Hacker News discussion