如何從用戶不應看到的狀態設計故障測試

I make Just Every, a set of free, open-source alternatives to subscription work software, so I spend a lot of time deciding what a test should check. Nothing in the method below needs our code. A worked example from one of our products is at the end.

Most test suites are organised around what the code does. I find a failure test easier to design from the other end: start with the wrong thing a user could see on screen.

Here's an exercise for it. Run it on a throwaway environment with test data.

1. Pick one visible claim. An operation shown as complete, or a status page showing everything healthy.
2. Write down the evidence that justifies it. A page that loaded isn't enough, because the write behind it can still have failed.
3. Take away one dependency at a time. The database, the cache, the mail provider, a stored fallback.
4. Check two things separately. What the user sees, and what the operator is told. A correct error page and a useful log line are different requirements.
5. Put the dependency back and repeat the action. Does the retry duplicate the work, or lose it?

Two habits make it work. Write the expected outcome down before you touch the code, otherwise it's easy to bend the expectation to whatever the code does today. And read the test's setup as carefully as its assertion. A check that never reaches the failure path passes forever and proves nothing.

An example: the status page that says "all clear"

Lamp is the open-source status page we make. One of its written promises is that a failed read never produces the all-clear. The test starts Lamp, removes a table from under the running database, opens the public page and checks that "All systems operational" appears nowhere. Then it takes away the page's way to find its saved copy and checks again. When the saved copy is available, the page serves it, labelled as the last known state.

The build also reads Lamp's list of promises and stops if any of them has no test, so a promise can't be added and left unchecked. Coding agents write the code against that list.

Promises change too

Lamp used to promise that a scheduled update goes live within 60 seconds. A stress run with 40 scheduled updates due in the same minute put the last one 238.5 seconds late. Lamp 1.4.0 promises 240 seconds, and its release record keeps the old promise, the new one and the measurement. I'd rather promise four minutes I've watched than sixty I haven't.

Lamp is MIT-licensed. The write-up with the test is at https://justevery.com/process and the rest of what we make is at https://justevery.com/.

Which failure state do you test first on a tool you're about to depend on?