手動測試員角色正在消亡：一位 QE 負責人不再為它辯護

In August, I told our dev team they could no longer hand a finished feature over to a QA department and wait for us to check it. They would be responsible for testing their own work before release and fixing problems that reached production.

I lead the Quality Engineering team at pdf.net, a subscription-based PDF editor. Proposing this change meant facing an uncomfortable question: if we weren’t checking other people’s features, what were we here for?

I started my career in manual testing and had spent years defending its value. But I stopped believing that a manual check at the end of the pipeline makes the product better. It turned out to cost more than it caught.

We used to spend our days checking tickets. Now we dig into production data, fix some bugs ourselves, and build a test suite the whole engineering team contributes to. Payments still get checked by hand, and so do hovers and animations. What’s gone is the queue of tickets waiting for QA.

By the way, the business hadn’t asked us to cut costs or reduce the team. A couple of months after the change, everyone in QA is still here, and releases still go out every working day.

We’re still early in the transition, but we already have a much better answer to what our QA team is here for.

“QA will check it.” I used to believe that too.

When I joined pdf.net as its first QA, there was no test environment and no automated test suite. While I built those, I also checked features by hand. When a ticket moved to “Test Needed,” I picked it up, and I either sent it back or cleared it for release.

This seemed like a sensible way to keep bugs out of production. Over time, though, I began to see three problems with it.

The first was the escape hatch. When there’s QA after you, you always have a way out. Not sure the feature works? QA will take a look. Didn’t check on mobile? QA will take a look. Didn’t think you might break the neighboring team? QA will find it.

And the better QA works, the less the developer thinks things through.

The second was scale. Developers started writing code with AI, and features arrived faster than we could check them. Releases waited in testing while tickets went back and forth between developers and QA. We hired two full-stack QAs who could do both manual testing and automation. That eased the workload but didn’t change the equation: every feature still needed someone’s time. Manual checking scales with hiring but code scales faster with tokens. While HR is still scheduling the first interview, AI has already opened another couple dozen PRs.

The third problem mattered most to me. During a conversation about our process, someone said that manual testing here was effectively invisible. It stung because I knew how much work went into a checked ticket, and how often I’d had to explain its value to the business.

Sure, we kept a record of everything we’d checked. But that record sits in a closed ticket: nobody rereads it, and it won’t check the feature again when someone else’s PR breaks it a month later. The next release needed the same work again, however well we’d done it the first time.

And yet that summer I was still saying there was “pretty much no way” we could handle all the new features without manual testing.

This time, QA won’t take a look.

Anyone could have proposed removing manual checks.

From management, this would sound like a budget cut. From developers, a way to get QA out of the way. We proposed it ourselves, and it was on us to make it work and find a more useful role for the team.

The formula we agreed on with all the engineers is simple: if a developer isn’t sure the feature works, the task isn’t done.

The August announcement made that official, and QA became Quality Engineering, or QE. It came with an FAQ and a thread where someone predicted a holy war. There was a small one :)

We’re a subscription product, so we kept one explicit exception: our QE team still checks major payment changes by hand, using real cards. Elsewhere, the feature’s author owns testing before release and the team owns any bugs that reach production. Where possible, we reproduce a defect with a test that fails on the current code, then fix it so the same test protects against a recurrence.

I expected resistance from developers over writing tests. Instead, once they owned the feature in production, tests became insurance. Nobody wanted to be the author of a bug that reached users, and a test run gave them a way to check their work before that happened.

Our developers have a preview environment for every PR, where they can run end-to-end tests and ask a browser agent to walk through a user scenario. They also have BrowserStack for checks on real devices. They still need to look at their work themselves; we don’t trust the agents to get everything right.

The author writes tests for the feature, and QE reviews them. We wrote guides covering product context, what belongs in the suite, and how tests should be structured. Most of the tests are now generated by AI, which makes those instructions especially important. A well-structured test can still check the wrong thing.

That’s where our review effort goes now. A manual check tells us whether a feature works today; a test we’ve reviewed can check it again on every merge. As tickets get closed and specs go stale, those tests also remain a working record of how the product is supposed to behave.

We have far more tests now, with developers contributing alongside QE. Quality is becoming part of how the whole team builds the product.

So what are we here for?

Calling ourselves Quality Engineers would mean very little if all we did was change our titles. Each member of the team took ownership of an area that needed engineering work, and our responsibilities now extend well beyond the testing queue.

We maintain the infrastructure and test architecture, review the tests engineers submit, and improve the rules that people and agents use to write them. A failing run should point to a real problem rather than send someone chasing a flaky test. We also own cross-team scenarios that are easy to miss when each developer is focused on their own feature. We still write end-to-end tests ourselves for those less obvious paths, just less often than we used to.

QE still has a role in release decisions. The release process is largely automated, but the on-duty quality engineer checks the required results and bot signals before merging.

More of our attention now goes to production: errors, logs, uptime and product analytics, and what normal looks like for each. When the fix is within our reach, we make it ourselves instead of creating a ticket for another team. When we need better visibility into what users experience, we can add the analytics events ourselves too.

Agents help us make that work manageable. They inspect signals, investigate possible defects and try to reproduce them; a separate agent challenges findings before we turn them into tickets. In the evening, another agent reviews where the agents themselves went wrong that day and changes one rule. A bot also triages failing runs on staging (I’ve written about that workflow separately).

People decide what deserves action and who should take it. The agents don’t assign work, post in shared channels or close issues on their own.

This also gives us a different conversation to have with the business. QE presents at demos alongside the product teams. We explain how many users a problem affected, what changed for them, and what it means in money. Access to product analytics lets us put errors, users and money into the same picture.

For a team whose work used to disappear into closed tickets, that gives us something tangible to show to the business.

Then production had a few things to say

There won’t be a pretty before/after chart. The transition is only a couple of months old, so we don’t have a full comparison yet. We’re still shipping new features every working day, and release quality hasn’t dropped.

In the first month, we brought teams dozens of real bugs our old process missed. These were problems we found after making time to examine production more closely, rather than relying on the scenarios we knew to check before release.

The editor was crashing when some users opened a document. After the fix, we saw no further cases on the new versions. Another crash affected older Safari versions and older iPhones: in that case, QE investigated and fixed it without handing a ticket to a product team.

For several hours one evening, some users couldn’t see Apple Pay, Google Pay or PayPal at checkout. The payment buttons were missing, but no errors were being reported.

A routine pass with our own files and devices hadn’t exposed those problems. Production signals did: errors and metrics.

The transition also exposed weaknesses in the system we were relying on. Our staging environment didn’t always match production: feature flags and experiments were configured differently, making test failures harder to interpret. When an experiment shipped without going through QE, most of the suite failed. Among those failures was a real bug that forced us to roll back the change. We needed staging to match production so we could trust what the tests were telling us.

The early AI-generated tests were another warning. They looked tidy and passed, but some barely tested anything useful. We tightened the guides and made review mandatory. We also had to be clear that more end-to-end tests weren't automatically better: anything we could check reliably at a lower level belonged there.

Some cases still appear only in production, often involving a rare browser, a particular payment method or an unusual sequence of actions. We review those signals every day and add regression coverage wherever we can. Monitoring is part of how we verify the product, and it doesn’t spare us the obligation to learn from a failure.

I thought we couldn’t do without it

The obvious objection is that humans catch things automated tests miss. I know they do. That summer, we caught a critical bug by hand in an experiment enabled for a small share of users. Hovers, animations, and timing still need eyes, and exploratory testing hasn’t disappeared.

Those checks matter, but I think the role built entirely around checking someone else’s finished feature is dying. If your value is walking through a checklist by hand, an agent will do it cheaper, and a developer faster, because they know what they changed.

Betting that “AI will never understand the user” is a bad strategy. A good one is to stop being the last line of defense and become the one who designs it.

For our team, that means understanding the product and the money, seeing risk before code is written, reading data, and writing the rules that people and agents work by. It demands more of us than accepting or rejecting a ticket, and gives us more scope to act on what we find.

We’ve taken the risk while this still looks like an experiment. I think almost everyone will work this way in a couple of years, though regulated industries (like fintech and healthcare) will take longer and need their own approach.

Earlier that summer, I’d said there was “pretty much no way” we could work without routine manual QA. Putting that belief to the test gave me the most useful red run of my career.

If your team still has a ‘Test Needed’ column, I’d like to hear what keeps it there.