數據治理中的可質疑智能體：記錄人類對結構化與非結構化系統中 AI 生成元數據的監督

Organizations deploying LLMs to classify operational data face a governance gap. Agents tag telemetry points, link warranty claims to service bulletins, and propose metadata for unstructured documents. When a human overrides an agent decision, most systems discard the agent's reasoning and store only the final value. You lose the audit trail, the correction context, and the signal needed to prevent classification drift.

A new ArXiv paper (2610.31553v1) introduces contestable governance: a pattern where agents produce scored, cited arguments for metadata decisions, stewards can accept or modify those arguments before commit, and a governance record retains the evidence, interventions, and outcome. The plumbing treats human corrections as first-class events, not silent overrides.

The Metadata Governance Problem

Industrial systems hold operational knowledge in two forms:

• Structured: telemetry streams, point lists, transaction logs, equipment hierarchies
• Unstructured: service manuals, inspection reports, communications, warranty claims

Governance programs manage these separately. Linking a telemetry point to its manual section, classifying a warranty failure mode, or tagging an HVAC zone requires domain specialists. Agents can propose classifications, but unreviewed proposals spread errors. A perfunctory approval leaves no record of what the reviewer actually checked.

The failure mode: agent-generated metadata drifts as equipment changes, terminology evolves, or edge cases accumulate. You discover the drift when downstream analytics break or compliance audits fail.

Contestable Governance Architecture

The pattern introduces three primitives:

1. Scored arguments: Agents produce metadata proposals with confidence scores and citations to source documents
2. Stewardship events: Human reviewers accept, reject, modify, or add arguments before commit
3. Governance records: Versioned logs that capture the agent's reasoning, the human's intervention, and the final decision

Workflow Shape

Agent Proposal → Triage → [High Confidence → Auto-Commit]
                        → [Uncertain → Steward Review → Governance Record]

High-confidence proposals (score above threshold) commit automatically but still generate a governance record. Uncertain proposals enter a review queue. The steward sees:

• Agent's proposed metadata value
• Confidence score
• Citations to source documents (manual sections, telemetry patterns, prior decisions)
• Conflicting arguments if multiple agents disagree

The steward can:

• Accept the proposal as-is
• Reject and provide an alternative with justification
• Modify the proposal and add supporting evidence
• Flag the decision as a precedent for future similar cases

Data Structures

The governance record schema:

@dataclass
class GovernanceRecord:
    record_id: str
    timestamp: datetime
    target_entity: str  # telemetry point ID, claim number, document hash

    agent_proposals: List[ArgumentProposal]
    steward_decision: StewardDecision
    final_metadata: Dict[str, Any]

    precedent_flag: bool
    version: int

@dataclass
class ArgumentProposal:
    agent_id: str
    proposed_value: Any
    confidence_score: float
    citations: List[Citation]
    reasoning: str

@dataclass
class StewardDecision:
    steward_id: str
    action: Literal["accept", "reject", "modify", "defer"]
    justification: str
    added_citations: List[Citation]

The record lives in a separate governance store, not inline with the operational data. You query it for audit trails, drift detection, and agent retraining.

Versioned Domain Glossary

Agents reference a versioned glossary that supplies:

• Model-specific terminology
• Acronym definitions
• Taxonomy mappings (e.g., Brick Schema classes for HVAC points)
• Deprecated terms and their replacements

When the glossary updates, you can replay governance records to detect which decisions would change under new definitions. This surfaces semantic drift before it propagates.

Implementation: HVAC Point Classification

The paper presents a reference implementation for classifying HVAC telemetry points using Brick Schema labels.

Input: A telemetry point with raw metadata:

{
  "point_id": "AHU-3-SAT",
  "description": "Supply air temp sensor zone 2",
  "unit": "degF",
  "equipment_parent": "AHU-3"
}

Agent task: Propose a Brick Schema class (e.g., brick:Supply_Air_Temperature_Sensor).

Workflow:

1. Agent queries the glossary for "supply air temp" and finds brick:Supply_Air_Temperature_Sensor
2. Agent checks equipment hierarchy: AHU-3 is an Air Handling Unit
3. Agent scores confidence at 0.92 based on term match and hierarchy consistency
4. Proposal auto-commits because score exceeds 0.85 threshold
5. Governance record logs the decision with citations to glossary entry and equipment hierarchy

Uncertain case:

{
  "point_id": "RTU-7-OAT",
  "description": "Outside temp",
  "unit": "degC",
  "equipment_parent": "RTU-7"
}

Agent proposes brick:Outside_Air_Temperature_Sensor with 0.68 confidence (ambiguous: could be brick:Outdoor_Air_Temperature_Sensor). Steward reviews, confirms the correct class based on equipment manual, and adds a citation. The governance record captures both the agent's uncertainty and the steward's resolution.

Triage Path and Precedent Gate

The triage logic routes proposals based on confidence score and similarity to prior decisions:

The precedent gate controls which verified decisions become training examples. Only decisions where:

• Steward explicitly reviewed the reasoning
• Confidence score was initially uncertain
• Decision resolved a conflict or edge case

This prevents auto-committed high-confidence decisions from polluting the precedent store with obvious cases.

Observability Primitives

You need metrics to detect when agent metadata quality degrades:

Drift indicators:

• Steward override rate (rejections / total proposals)
• Confidence score distribution over time
• Citation staleness (age of referenced documents)
• Precedent match rate (how often new cases resemble stored precedents)

Alerting thresholds:

• Override rate exceeds 15% in a 7-day window
• Mean confidence score drops below 0.75
• More than 20% of citations reference documents older than 6 months

Dashboard queries:

-- Detect classification drift by equipment type
SELECT 
  equipment_type,
  COUNT(*) FILTER (WHERE action = 'reject') AS rejections,
  COUNT(*) AS total_proposals,
  AVG(confidence_score) AS mean_confidence
FROM governance_records
WHERE timestamp > NOW() - INTERVAL '30 days'
GROUP BY equipment_type
HAVING COUNT(*) FILTER (WHERE action = 'reject') / COUNT(*) > 0.15;

Warranty Failure-Mode Decision Example

The paper's second use case links warranty claims to service bulletins and telemetry patterns.

Input: A warranty claim with fields:

{
  "claim_id": "WC-2024-8832",
  "failure_description": "Compressor short cycling",
  "equipment_serial": "RTU-2024-4471",
  "claim_date": "2024-09-15"
}

Agent tasks:

1. Classify failure mode (e.g., "electrical", "mechanical", "controls")
2. Link to relevant service bulletins
3. Cite telemetry patterns (if available) showing short cycling behavior

Agent output:

• Failure mode: "controls" (confidence 0.71)
• Cited bulletin: SB-2023-09 "Compressor Cycling Due to Faulty Pressure Transducer"
• Cited telemetry: 47 start/stop events in 2-hour window on 2024-09-14

Steward review:

• Accepts "controls" classification
• Adds citation to revised bulletin SB-2024-02 (supersedes SB-2023-09)
• Marks decision as precedent for future compressor cycling claims

The governance record now links the claim, the telemetry evidence, and the correct bulletin version. When SB-2024-02 is revised again, you can query governance records to find all claims that referenced it.

Preventing Classification Drift

The feedback loop:

1. Governance records accumulate steward corrections
2. Precedent store indexes high-value decisions
3. Agents query precedents during proposal generation
4. Glossary updates trigger replay of affected records
5. Drift metrics surface when agent performance degrades

Retraining trigger: When override rate exceeds threshold, you can:

• Fine-tune the agent on precedent store examples
• Update the glossary with new terms from steward justifications
• Adjust confidence thresholds for specific equipment types

The key: you have structured data (governance records) instead of unstructured feedback (emails, tickets, manual corrections in spreadsheets).

State Management Trade-offs

The paper recommends separate governance store with periodic metadata snapshots. You keep the operational database clean and query the governance store only for audits and retraining.

Security Boundaries

Access control layers:

• Agents can propose but not commit
• Stewards can commit within their domain (e.g., HVAC specialist cannot override electrical classifications)
• Admins can mark decisions as precedents
• Auditors have read-only access to governance records

Citation integrity: All citations include document hashes or version identifiers. If a cited document changes, the governance record flags the citation as stale.

Steward identity: Each governance record logs the steward's identity and timestamp. You can audit who approved which decisions and detect patterns (e.g., one steward consistently overriding a specific agent).

Deployment Shape

Minimal stack:

• Agent service (LLM API + glossary lookup)
• Governance store (PostgreSQL with JSONB for argument structures)
• Steward UI (review queue + citation viewer)
• Triage service (confidence scoring + precedent similarity search)

Integration points:

• Operational database (telemetry, claims, equipment hierarchy)
• Document store (manuals, bulletins, inspection reports)
• Glossary service (versioned terminology + taxonomy mappings)

Scaling considerations:

• High-confidence auto-commits bypass the review queue
• Precedent similarity search uses vector embeddings (e.g., sentence transformers on decision justifications)
• Governance records partition by equipment type or domain

Likely Failure Modes

Steward fatigue: If too many proposals land in the review queue, stewards rubber-stamp decisions. Mitigation: tune confidence thresholds, improve agent prompts, surface precedent matches to reduce cognitive load.

Glossary staleness: Agents cite outdated terms, stewards correct them, but glossary never updates. Mitigation: automated glossary update proposals when override rate for specific terms exceeds threshold.

Citation rot: Linked documents move, get revised, or deleted. Mitigation: periodic citation validation job, flag stale citations in governance records.

Precedent pollution: Auto-committed decisions leak into precedent store. Mitigation: strict precedent gate (only steward-reviewed, initially-uncertain decisions qualify).

Conflicting precedents: Two stewards resolve similar cases differently. Mitigation: precedent conflict detection, escalation to domain lead for reconciliation.

Technical Verdict

Use contestable governance when:

• You deploy agents to classify or tag operational data at scale
• Compliance or audit requirements demand evidence trails
• Domain specialists are scarce and their corrections are valuable training signal
• You need to detect and prevent metadata drift in production systems

Avoid when:

• Classification decisions are low-stakes and errors are cheap to fix
• You lack domain specialists to review uncertain proposals
• Your operational database already has strong versioning and audit logs
• Agent accuracy is high enough that human review adds no value

The pattern shines in regulated industries (energy, manufacturing, healthcare) where metadata errors propagate into compliance failures. It adds complexity: you maintain a separate governance store, build a steward UI, and tune confidence thresholds. The payoff is structured feedback that improves agent performance and prevents silent drift.

If you are already logging agent decisions, contestable governance is the next step: capture not just the outcome but the reasoning, the human intervention, and the evidence. You turn corrections into training data instead of lost context.

Source Links

• ArXiv Paper (2610.31553v1)
• PDF