資深測試人員如何循序學習 AI 測試

Short answer: learn it in this order — the non-determinism mindset, prompts as test inputs, evals (the new assertion), LLM/RAG failure modes, then verifying AI-written tests with mutation testing. Skip the math-heavy ML theory; you're testing behavior, not training models.

I keep seeing the same post on r/softwaretesting: "10 years of manual and automation under my belt, and the AI stuff makes me feel like a junior again. Where do I even start?" The honest answer is that you're closer than you think. Maybe 70% of what makes you a good tester transfers directly. The other 30% is a handful of new ideas, and they stack in a specific order. Learn them out of order and it feels like chaos. Learn them in order and each one clicks.

Here's the order I'd follow.

1. Get the one mindset shift first: non-determinism

Everything new starts here. In normal testing, one input has one correct output, so you assert equality. In AI testing, one input has a set of valid outputs — ask the same question twice and the wording changes, both answers fine. That single fact breaks every exact-match assertion you've ever written and is why your first naive AI tests will flake.

Don't move on until this feels natural, because the rest of the roadmap is just tools for dealing with it. If a test flaps on identical input, your instinct should flip from "the system is flaky" to "I'm asserting the wrong thing."

2. Learn prompting — as a tester, not a hobbyist

Everyone treats prompting as a party trick. For you it's controlling the test input. A vague prompt is an uncontrolled variable; you can't tell a real regression from your own sloppy setup. Spend a week writing prompts that pin down format, scope, and constraints, and reading back exactly what the model did with them. You're not learning to be clever. You're learning to make the input reproducible enough to test against.

This is the step most testers skip, and it's why their AI tests feel unreliable later.

3. Evals — the assertion for the AI era

This is the real new skill, and it's where I'd spend the most time. An eval replaces "did the output equal X?" with "is this output acceptable?" Three flavors, in increasing effort:

• Property checks — rules true for every valid answer. No PII leaked, no invented dollar figures, output parses as JSON. Deterministic, cheap, catch the scary failures.
• Reference/similarity — score the answer against a known-good one with embeddings or keyword coverage, assert a threshold, not equality.
• LLM-as-judge — have a model grade against a rubric. Powerful for open-ended text, but it's non-deterministic too, so pin its model and pair it with hard property checks.

def test_support_reply():
    out = bot("my card was declined")
    assert out.intent == "billing"              # property: deterministic
    assert "$" not in out.text                   # property: no invented amounts
    assert judge(out.text, rubric="acknowledges, asks one question, no blame") >= 0.8

If you learn one thing from this whole list, learn to write evals. It's the skill job posts are circling around without naming.

4. The LLM/RAG failure modes you'll actually hit

Now get specific about how these systems break, because it's not how normal software breaks — and how to tell a real bug from the model just drifting:

• Hallucination — confident, fluent, completely made up. Your test is a grounding check: did the answer stick to the provided context?
• Retrieval failure (RAG) — the model's fine, but it fetched the wrong documents. Test retrieval separately from generation, or you'll debug the wrong half for a day.
• Prompt injection — user input that hijacks the instructions. This is the new security test, and it's yours now.

You don't need to know how a transformer works to test any of this. You need to know what wrong looks like, which is exactly what testers are good at.

5. Use AI to write tests — then prove the tests work

You'll let Copilot or Claude generate test suites. Fine. But generated tests have a specific trap: they tend to assert that the code does what it currently does — bugs included. A green suite that was written against buggy code is just the bug, notarized.

The fix is mutation testing: deliberately break the code and check your tests scream. If you flip a > to >= and every test still passes, those tests prove nothing. Run this on AI-generated suites especially — it's how you actually earn confidence in AI-written tests. This is the whole point of the book, and the free lab is a 10-minute demo of it — a suite that's green against four real bugs.

What to skip

You do not need: the math behind training, building models, PyTorch, or a data-science course. You're testing behavior at the boundary, the same place you always have. Picking up ML theory first is the most common way people stall for three months and learn nothing testable. Skip it until something forces you to care.

Realistically this is a few focused weeks, not a bootcamp — and you keep every bit of your existing automation, CI, and bug-sense. Don't let anyone sell you "AI testing" as a reset. It's an extension.

FAQ

Do I need to learn Python?
If you already automate, you're set — most eval and LLM tooling is Python-first, but it's the same testing you do now with different assertions.

Is this going to replace traditional QA?
No. It sits on top of it. CI, coverage, API and E2E testing all still apply; AI adds non-determinism and a new set of failure modes to cover.

Where does the book fit in step 3 and 5?
Those two steps — evals and verifying AI-written tests — are the spine of AI Testing in Practice. It walks the full version with runnable examples, so you're not piecing it together from scattered blog posts.