Vibe coding 測試指南:如何在用戶使用前檢查應用程式
Vibe coding testing: check your app before users do
文章提出一套檢查 AI 工具編寫的應用程式的方法,建議為核心操作建立可執行的測試,並在上線前以用戶的方式完成真實任務。作者亦建議逐項修改後重跑測試,讓全新工作階段檢查程式,並先在預覽版本試用;上線後收集用戶遇到的錯誤,為每項修正留下測試。
Originally published at pantheontable.com/blog/vibe-coding-testing-check-your-app-before-users-do
Andrej Karpathy vibe coded a website and felt 80% done. By his own account, it was a bit closer to 20%.
If you built an app with an AI tool and it seems to work when you click through it, this is for you. Below is what Karpathy, David Heinemeier Hansson, John Carmack, Joel Spolsky, Simon Willison, Jeff Atwood and W. Edwards Deming said about bugs and testing, with every quote linked to its source, and step lists you can hand straight to your AI tool.
The short answer
Clicking around proves only the paths you happened to try, so vibe coding testing means giving the AI a check it can run and then using the app for a real task the way a customer would. Ask the AI to write tests for the core actions, show you the command and its output, and let a fresh session review the changes. Try every change on a preview copy before it goes live, and once real users arrive, collect their errors so you know when something breaks.
Why clicking around is not enough for vibe coding testing
You built an app with an AI tool, you clicked through it, and it seemed to work. That feeling is real, and it is also the trap. A quick click-through follows the paths you expect, while your users will take the ones you never thought of: the empty form, the double click, the phone on a weak connection, the second account.
"Seeing a new website materialize so quickly is a strong hook. I felt like I was 80% done but (foreshadowing...) it was a bit closer to 20%."
Andrej Karpathy, Vibe coding MenuGen (2025-04-27)
He wrote this about a website he built by vibe coding: the part that looks finished on screen is the small part.
What happened (2025): Testing AI Vibe Coding. DHH attempted to use AI 'vibe coding' to build real systems but found it failed at scale. While it could create a functional-looking veneer, it often broke multiple things when trying to fix a single bug, similar to a junior programmer.
Lesson: Current AI tools can handle simple tasks but struggle with complex system integrity, requiring human expertise to verify the output.
David Heinemeier Hansson, Transcript for DHH: Future of Programming, AI, Ruby on Rails, Productivity & Parenting | Lex Fridman Podcast #474 - Lex Fridman (2025-07-12)
The pattern to watch for is not one bug, but a fix that quietly breaks something you had already checked.
Put those records together and the lesson for you is plain. A working screen is not a working app, and a fix from the AI can move a bug instead of removing it. So your testing has to be repeatable: something you or the AI can run again after every change, not a memory of what worked yesterday.
What to ask the AI to check
The most useful thing you can give an AI coding tool is a way to know whether it succeeded. Without one, it stops when the work looks finished to it, and you end up as the only test your app has.
Checked fact: Anthropic's best-practice page says the tool stops when the work looks done; if it has nothing to run that says pass or fail, you become the test and every mistake waits until you notice it.
Anthropic, Claude Code best practices; GitHub Docs, Copilot best practices; Lovable and Replit documentation, checked 2026-11-18
Checked fact: The check can be a test file, a build, a linter or a browser screenshot next to the design; with it the tool can find and fix its own mistakes.
Anthropic, Claude Code best practices; GitHub Docs, Copilot best practices; Lovable and Replit documentation, checked 2026-11-18
Principle: Automate Quality Assurance with Agents. AI agents are often more effective than humans at finding race conditions and bugs in production code through continuous review.
David Heinemeier Hansson, Transcript for DHH: Future of Programming, AI, Agentic Engineering, Vibe Coding & Linux | Lex Fridman Podcast #501 - Lex Fridman (2026-08-26)
DHH's record points the same way from the other side: an agent reviewing code is often better than a person at hunting race conditions and other bugs.
"A large fraction of the flaws in software development are due to programmers not fully understanding all the possible states their code may execute in."
John Carmack, Wikiquote: John D. Carmack
This is the list to hand the AI: not just the happy path, but every state the app can be in.
- Ask the AI to list the core actions of your app (sign up, log in, pay, save, delete) and write an automated test for each one.
- Ask it to list the states each screen can be in, such as empty, loading, failed, very long input or a user without permission, and to test the ones that matter.
- Ask it to look for race conditions, such as the same button pressed twice or two tabs saving at once.
- Ask it to run the tests and paste the exact command and its output, not just a summary saying everything passed.
- Ask it to explain in plain words any part of the code you do not understand before you accept it.
How to test code the AI wrote when you are not a developer
Here is the hard part for a non-developer: the records do not pretend the AI can be left alone.
Principle: Expertise Validates AI Output. AI tools are most effective when the user is already an expert who can judge the quality of the output; otherwise, they are a gamble.
Arvid Kahl, Addressing My Weaknesses (2024-03-29)
Kahl's point is uncomfortable but useful: if you cannot judge the output yourself, you need other ways to judge it.
"Right now the agents are like interns. You still have to be in charge of aesthetics, judgment, taste, and oversight."
Andrej Karpathy, Sequoia Ascent 2026 summary (2026-04-30)
An intern can do real work, and an intern's work still gets checked before it reaches a customer.
You cannot become an expert overnight, but you can borrow judgment. Make the tool explain itself, get a reviewer that did not write the code, and keep each change small enough that you can see what moved.
Checked fact: GitHub's Copilot best practices say to validate all suggested code, to ask for an explanation of code you do not understand before using it, and to review it for function, security, readability and maintainability.
Anthropic, Claude Code best practices; GitHub Docs, Copilot best practices; Lovable and Replit documentation, checked 2026-11-18
Checked fact: A review by a fresh agent that never saw the conversation (the /code-review command in Claude Code) is not swayed by the reasoning that produced the code.
Anthropic, Claude Code best practices; GitHub Docs, Copilot best practices; Lovable and Replit documentation, checked 2026-11-18
Checked fact: Cursor's quickstart says to check the agent's changes in the diff view after it finishes and then run the project's existing checks.
Anthropic, Claude Code quickstart and best practices; Cursor, Lovable, Replit and GitHub quickstarts, checked 2026-11-18
- Ask for one change at a time, and test it before asking for the next.
- After each change, open the diff view or ask the tool which files changed and why, and question anything outside your request.
- Before launch, start a fresh session and ask it to review the whole project for bugs and security problems, without telling it what you suspect.
- If your corrections keep failing, stop and start a clean conversation with a clearer request.
Use your app for a real task, the way a customer would
What happened (2001): Finds 45 CityDesk bugs in one Sunday. About three weeks before CityDesk was to ship, Spolsky took a build home and tried to build a simple site with it, finding 45 bugs in about two hours. On Monday he told the team and asked everyone to build at least one serious site with CityDesk. The bugs were fixed and missing features were added.
Lesson: Before shipping, have the whole team use the product for a real task; it surfaces what formal testing misses.
Joel Spolsky, What Is The Work Of Dogs In This Country (2001-05-05)
Nothing exotic found those bugs: just Spolsky taking a build home and trying to do a real job with the product.
What happened (1996): Solving Latency with QuakeWorld. The original networking code worked well on LANs but lagged on dial-up because it waited for server confirmation. Carmack released QuakeWorld with client-side prediction, which allowed the game to react instantly to player input even with high latency.
Lesson: Test your product in the same environment as your customers, not just on high-end internal systems.
John Carmack, Wikipedia: Quake (video game)
The code worked well on fast local networks and lagged on dial-up. Your phone on mobile data is your customers' condition.
If you want to test an app before launch with no technical skill at all, this is the method. Pick the real job your app exists to do and do it from start to finish as a stranger would: a new account, real data, a phone instead of your laptop, a slow connection. Then ask a friend to do the same while you watch without helping. Every place they hesitate is a bug, even if nothing crashed.
Checked fact: Lovable's quick start has a test step before publishing: click through the buttons and links and check the mobile view. Replit's guide says to use the app like a visitor and compare it with your success criteria.
Anthropic, Claude Code best practices; GitHub Docs, Copilot best practices; Lovable and Replit documentation, checked 2026-11-18
- Write down the job your app is for in one sentence, then do it from a fresh account.
- Repeat it on a phone, on mobile data and in a different browser.
- Click every button and link, and fill every form with wrong, empty and very long input.
- Have someone who has never seen the app try it while you stay silent, and note each stumble.
- Give each problem to the AI one at a time, with what you did, what you expected and what happened.
Test changes on a preview copy, not the live site
Once people use your app, every change you ask for lands on them. The builders in our library see that differently, and both views teach something.
Principle: Automate deploys to staging first. Willison advises gating every deploy on passing tests and rolling out automatic deployment to a staging instance first, switching to production only once you trust the pipeline.
Simon Willison, How to set up world-class continuous deployment using free hosted tools
Willison wants a gate: nothing reaches production unless the tests pass, and a staging copy gets it first.
Principle: Careful Coding as Testing. When working without a staging environment, the developer must develop the discipline to code more carefully to avoid breaking the live system.
Pieter Levels, Transcript for Pieter Levels: Programming, Viral AI Startups, and Digital Nomad Life | Lex Fridman Podcast #440 - Lex Fridman (2024-08-20)
Levels speaks to working without that safety net, and his record is honest about the price: the discipline has to come from you.
For someone new to code, the gated route is the safer one. Vercel, for example, keeps preview deployments apart from the live site, so you can click through the new version before your users see it. Keep your project in Git as well, so that when a change goes wrong you can return to the last version that worked.
Checked fact: Vercel separates a preview deployment (for testing without touching the live site) from the production deployment (the live site on your domain).
MDN Web Docs (Mozilla contributors), 'Publishing your website'; Vercel documentation, 'Deploying to Vercel', checked 2026-11-18
Checked fact: The docs state that any earlier version of the project can be recovered at any time.
GitHub Docs, 'About Git' and 'About GitHub and Git'; Anthropic, Claude Code documentation, checked 2026-11-18
After launch, make sure you know when something breaks
"If you're going to move fast and break things, you need to know when things have broken."
Simon Willison, How to set up world-class continuous deployment using free hosted tools
Testing does not end at launch; it changes shape.
What happened: Collects field crash dumps at Juno. While Spolsky worked at Juno, the team installed a handler that dumped the stack to a log whenever the program crashed and uploaded it on the next connection. During betas they collated these logs into the bug database.
Lesson: Automatically collecting crash data from real users surfaces bugs you would never find in testing.
Joel Spolsky, Hard Assed Bug Fixin (2001-07-31)
Real users find bugs your tests never will, but only if their crashes reach you.
Principle: Weigh customer bugs over tester bugs. Atwood argues testers aren't real users, so he gives a customer-reported bug ten times the weight of one found by a tester.
Jeff Atwood, Not All Bugs Are Worth Fixing (2025-06-05)
So when a real customer reports a bug, it goes to the top of the list.
Ask the AI to add error reporting to your app, so that crashes and failed requests are recorded along with what the user was doing, and to send you an alert when they happen. Give early users an easy way to report a problem from inside the app. Then treat what they send as your most valuable test results.
Make every fix leave a test behind
"Quality comes not from inspection, but from improvement of the production process."
W. Edwards Deming, Wikiquote: W. Edwards Deming
Checking at the end will always miss things; the cure is in how the work is done.
Podcast summary: Episode 386 | Balancing Feature Development with Marketing, the Cost of Technical Debt, and More Listener Questions. Among its points is the habit that turns that idea into a daily routine: add a test with every bug fix. (the episode's official page)
Here is how that looks with an AI tool. When a user reports a bug, first ask the AI for a test that fails because of it, then ask for the fix, then run all the tests again. The bug cannot come back silently, and your collection of tests grows from real problems instead of guesses. Over time, the tests become the memory your vibe-coded app was missing.
Checked fact: Lovable's quick start says to iterate with one change at a time; Replit's guide says to ask for one focused improvement and then retest.
Anthropic, Claude Code quickstart and best practices; Cursor, Lovable, Replit and GitHub quickstarts, checked 2026-11-18
Questions people ask
How do I test code the AI wrote if I can't code?
Give the AI a check it can run: ask it to write tests for your app's core actions and to show you the command and its output. Then use the app for a real task yourself, on a phone, and have a fresh session review the code before launch.
What should I test before launching my app?
The core job your app exists for, done from a new account, on a phone and in another browser; every button, link and form with empty, wrong and very long input; and anything that handles money or personal data.
Can the AI test its own code?
Partly. It can write and run tests and is good at hunting bugs such as race conditions, but a message saying everything passed tells you little without the command and its output, and a fresh reviewer is not swayed by the reasoning that wrote the code.
Do I need a staging server or preview deployment?
It is the safer route for a beginner: try each change on a preview copy and send it to the live site only when it works. Without one, you have to work very carefully, because every mistake lands on real users.
What should I do when a user reports a bug?
Put it at the top of the list, ask the AI for a test that reproduces it, then the fix, and rerun all the tests so the bug cannot return unnoticed.
Bring your own question
If your launch is close and you are not sure where to start, bring a question like "My app launches soon and my time is short. What should I test first, and what can wait?" to The Pantheon Table. Your business and investing questions are answered there with the real words of world-class masters: https://pantheontable.com/blog/vibe-coding-testing-check-your-app-before-users-do?utm_source=devto&utm_medium=social&utm_campaign=blog
The masters' words are quoted from public records, each with its source. None of them wrote, reviewed or endorses this guide.
Photo: Gladwin Analytics, CC BY 3.0.
Not affiliated with or endorsed by Andrej Karpathy or John Carmack.
來源:DEV Community · dev.to


