測試 AI 應用能否在後續步驟保留用戶修正的方法
Test Whether a User Correction Survives the Next AI Step
這份指南提出一套測試方法，檢查 AI 應用能否在用戶修正錯誤抽取值後，讓摘要及下游動作沿用已確認資料。測試應分開核對資料持久化、重新生成及下游動作，並記錄版本資訊和預期與實際欄位值。文中案例屬虛構方案，並非已執行的基準測試，且不應以真實客戶訊息或預訂驗證。
A model extracts the wrong date. The user edits it. A later summary brings the wrong date back.
This is a useful failure sequence to test in an AI application. It crosses model output, user input, application state and any downstream action.
The following example is fictional. It describes a proposed test approach, not results from an executed benchmark.
Establish the state contract
For a booking assistant, distinguish extracted suggestions from user-confirmed values. Specify which state a summary or dispatch worker must read and which events can invalidate confirmation.
A prompt asking the model to respect edits may help guide generation. The application still needs to enforce its rules when saving data or preparing an action.
Define how conflicts are resolved before writing assertions. Which fields may be regenerated? Does editing a recipient require renewed approval? Can a stale client overwrite a newer correction?
Build a reproducible correction case
Use a permitted fixture with a known date mismatch. Where appropriate, inject the incorrect extraction in a controlled test environment so this test exercises the correction path consistently. Keep model extraction quality tests separate; otherwise the absence of an extraction error can prevent the recovery path from being exercised.
Through the user interface, correct the date and continue the workflow. Inspect the values presented to the user and the relevant persisted state using the test environment's supported checks.
Inspect downstream consumers
Check the saved booking, generated summary and proposed external action. Include the version or revision information needed to identify which state each component used.
Do not send real customer messages or create real bookings simply to verify a test. Use isolated dependencies or approved test accounts appropriate to the integration.
Add cases that challenge ordering
| Case | Requirement to clarify |
|---|---|
| Regenerate after correction | Which user-confirmed fields must remain unchanged |
| Two tabs edit the same draft | How stale versions and conflicting edits are handled |
| Background generation finishes late | Whether its output can replace newer state |
| Recipient changes after approval | Whether the action requires fresh approval |
| Retry after a timeout | How uncertain external outcomes are reconciled |
| Permission changes during the workflow | Which checks apply at save and dispatch time |
The exact expected behaviour depends on the product. Avoid treating a table of scenarios as a universal specification.
Separate observations in the report
Report initial extraction behaviour, persistence correctness, regeneration behaviour and downstream action correctness separately. A single overall pass rate can hide a broken transition.
For generative steps, state the configurations and repetitions used. A deterministic state test and a repeated model-output evaluation answer different questions. Keep both their findings and limitations visible.
Make failure evidence useful
Capture the relevant revision identifiers, transition and expected versus observed field values. Protect private information and avoid dumping complete conversations into unrestricted test logs.
The goal is a failure report another engineer can investigate: which correction was made, which component read outdated state and where it appeared again.
Acadify AI Lab supports scoped LLM evaluation, AI testing and developer feedback. Acadify Solution builds custom software and AI applications.
Where have you seen user edits get lost: regeneration, persistence or a downstream worker?
Disclosure: written by an AI agent for Acadify Solution / Acadify AI Lab.
來源：DEV Community · dev.to