跳到正文
DEV Community· Ujjwal Dubey·· 4 天前AI 評分58

評估 AI 工具前,如何從工單匯出資料建立真實案例分層樣本

Build a Stratified Sample of Real Cases From a Ticket Export Before You Evaluate Any AI Tool

AI 導讀

Ujjwal Dubey提供一段 Python 腳本,從工單或查詢匯出的 CSV 按問題類型及類別抽取分層樣本,建立評估 AI 工具的案例集。腳本以固定隨機種子抽樣,並按類別輪流選取案例;作者建議手動加入預期結果,讓候選工具在同一批案例上以草稿模式測試和評分。

正文

Ujjwal Dubey

Vendors love to demo on clean examples. The best defence is your own test set, picked fairly from real history. That set is the backbone of any honest attempt to audit a business workflow with real data before buying AI.

Here is a small script that builds one from a ticket or enquiry export.

Input

Any CSV with an id, a category and a few signals of trouble. For example:

id,category,reopened,replies,minutes_to_first_reply
1001,refund,0,2,14
1002,delivery,1,6,310
1003,refund,0,1,9

The script

import csv, random
from collections import defaultdict

random.seed(7)  # repeatable sample

def bucket(row):
    if int(row["reopened"]) or int(row["replies"]) >= 5:
        return "awkward"
    if int(row["minutes_to_first_reply"]) > 240:
        return "slow"
    return "ordinary"

def sample(path, per_bucket=None):
    per_bucket = per_bucket or {"ordinary": 10, "awkward": 6, "slow": 4}
    groups = defaultdict(list)
    with open(path, newline="") as f:
        for row in csv.DictReader(f):
            groups[(bucket(row), row["category"])].append(row)
    picked = []
    for b, n in per_bucket.items():
        keys = [k for k in groups if k[0] == b]
        i = 0
        while n > 0 and any(groups[k] for k in keys):
            k = keys[i % len(keys)]
            if groups[k]:
                picked.append(groups[k].pop(random.randrange(len(groups[k]))) | {"bucket": b})
                n -= 1
            i += 1
    return picked

if __name__ == "__main__":
    rows = sample("tickets.csv")
    with open("eval_set.csv", "w", newline="") as f:
        w = csv.DictWriter(f, fieldnames=list(rows[0].keys()))
        w.writeheader(); w.writerows(rows)
    print(len(rows), "cases written to eval_set.csv")

Why it is built this way

  • Round-robin across categories so one busy category does not crowd out the rest.
  • Separate buckets for awkward and slow cases, because that is where tools fail.
  • A fixed seed so you and the vendor test on the same set.

What to do with the set

Add an "expected outcome" column by hand. Then run every tool you are considering on the same cases, in draft mode, and score the results. Keep the set private so nobody tunes against it.

I work at NxFlowAI, and we use a set like this after our 72-hour audit. For more posts on evaluation and workflows, see the NxFlowAI blog index.

來源:DEV Community · dev.to