HelloAI：相同事故筆記、兩種提示詞，一份草稿保留錯誤，另一份將其標示出來

Quick note first: I do marketing for HelloAI (tryhello.app), and this is a test I ran on its free plan. Everything below is fictional incident data I wrote, so treat the output as a draft that needs a human review, not a real postmortem.

Writing a postmortem from messy notes is a chore most teams know. So I gave a chat app a set of rough notes and asked for a blameless postmortem. I ran it twice, with two different prompts, and I'll show both in order.

The notes (fictional)

Tue 14:02 deploy of v2.3 went out. 14:10 error rate on checkout jumped to 12%. 14:18 on-call paged (alert fired 8 min late because threshold was set to 15%). 14:31 rolled back, errors dropped by 14:35. Cause: new config flag defaulted to off in prod but on in staging, so a null payment-provider timeout was used. 37 minutes of impact, about 210 failed checkouts. Fixes: add prod/staging config diff check to CI, lower alert threshold to 5%, add a canary step.

I wrote these quickly. Looking at them afterwards, two things don't hold up: 14:02 to 14:35 is 33 minutes (and 14:10 to 14:35 is 25), not 37, and a 12% error rate is below a 15% alert threshold, so "fired late because the threshold was 15%" doesn't explain much.

Run 1: "just write it"

Prompt: "Turn these messy notes into a blameless incident postmortem I can paste into a doc." plus the notes, plus a list of sections (summary, impact, timeline table, root cause, what went well, what went badly, action items with owners TBD) and "Do not invent any facts beyond the notes."

In about 15 seconds I got an editable document, around 260 words, with every section I asked for. Times, percentages, the 210 count and the three fixes were carried over correctly, and it added no new facts. Owners were left as TBD.

But it repeated "37 minutes" without comment, and it wrote that the 15% threshold meant the alert fired late after a 12% jump, which glosses over the inconsistency above. I hadn't asked it to check, so that is a fair result, but it's the kind of thing a draft can quietly hide.

Run 2: "check the notes first"

New chat, same notes, different prompt: "Before writing anything, check these incident notes for inconsistencies or unsupported numbers and list them. Then write a blameless postmortem draft that does not paper over them." (plus "Owners TBD. Do not invent facts.")

It listed these, and the first two are the ones I found myself:

• The 37 minutes doesn't match any interval in the notes: 14:02 to 14:35 is 33, 14:10 to 14:35 is 25.
• A 12% error rate is below a 15% threshold, and the notes don't say what the threshold measured or how the alert fired.
• "Lower alert threshold to 5%" doesn't name a metric or a duration condition.
• "About 210" failed checkouts has no stated basis.
• It isn't explained how the flag state led to a null payment-provider timeout.

The draft that followed carried these as open questions ("What interval does the reported 37-minute impact duration cover?") instead of smoothing them over.

One thing I'd flag: in the action items it changed my note "lower alert threshold to 5%" into "review and adjust the alert threshold, including the metric and any duration condition". That's reasonable given its own concern, but it is a change to a fix I had stated, so you'd want to restore the original wording if that was the decision.

What I take from it

Same notes, same free plan: the plain "write it" prompt gave a clean draft that kept my mistakes, and the "check first" prompt surfaced them and wrote around them. That is two runs, one set of notes I wrote, so it says nothing about how reliable this is in general. I'd still read any generated postmortem against the real timeline before sharing it.

If you try something similar, ask for the inconsistency check explicitly and keep your original notes next to the draft.