生成式 AI 開發中，提示詞約佔 20%，其餘 80% 是工程、數據與營運

Last spring I built a working generative AI feature in a weekend: a support assistant that read a customer's question, pulled context from our help docs, and wrote a friendly, accurate answer. The demo went great. Everyone nodded.

Shipping it took three more months.

Not because the prompt was wrong. The prompt barely changed. The time went into everything around it: cleaning the knowledge base, building an evaluation set, handling the cases where the model confidently made things up, keeping costs from spiraling, and wiring it into systems that had never heard of an LLM.

I've watched several teams go through the same arc, and I'm convinced of a rough rule: prompting is about 20% of generative AI development. The other 80% is engineering, data, and operations work that never shows up in a demo. Here's what that 80% looks like.

Why the demo lies to you

A demo is a best-case path. You pick the inputs, you know what the model handles well, and nobody asks a question you didn't anticipate. Production is the opposite. Real users type half sentences, paste entire emails, switch languages, and probe edge cases whether they mean to or not.

The gap between demo and production isn't a prompt-quality gap. It's a reliability gap. A model that's right 90% of the time looks magical on stage and unacceptable in a customer-facing workflow, where the other 10% becomes support tickets and refunds. Closing that gap is the job, and almost none of it is prompting.

The data work nobody budgets for

Most useful generative AI features sit on top of your own data: documents, tickets, product catalogs, policies. In our case, the help docs contained contradictory articles and three versions of the refund policy. The model happily retrieved all three and blended them into an answer that matched none of them.

So the real work looked like this:

• Deduplicating and versioning source content
• Deciding what's authoritative when documents disagree
• Chunking documents so retrieval returns coherent passages, not sentence fragments
• Adding metadata (product, region, date) so retrieval can filter before the model sees anything
• Building a refresh pipeline so the index doesn't go stale a month after launch

Retrieval quality sets the ceiling on answer quality. When the output is bad, look at what went into the context window before you touch the prompt.

Evals: what separates a project from an experiment

The biggest shift in how I work was treating evaluation as a first-class deliverable. Without an eval set, every change is a vibe check. You tweak the prompt, try five examples, and ship. Two weeks later you find that your improvement broke an entire category of questions, and you can't say when.

A basic setup doesn't need to be fancy:

1. Collect 100 to 200 real or realistic inputs covering common cases, edge cases, and known failures.
2. For each, define what good looks like: facts that must appear and things that must never be said.
3. Run every change (prompt, model, retrieval settings, chunk size) against the set.
4. Track scores over time and treat a regression like a failing test.

Use automated checks where you can, such as valid JSON, required fields, or a citation present, and model-based or human grading where you can't. Model graders have their own biases, so spot-check them against human judgment.

The payoff is confidence. You can swap models or change retrieval strategy and know within minutes whether things improved or worsened.

Guardrails and graceful failure

Every generative AI system will eventually produce something wrong, off-brand, or unsafe. What matters is what happens next. Plan for failure at several layers:

• Input checks: catch prompt injection attempts, out-of-scope requests, and sensitive data before they reach the model.
• Output checks: validate structure, verify claims against retrieved sources, and filter policy violations.
• Fallbacks: when confidence is low, hand off to a human or say "I'm not sure" instead of delivering a polished guess.
• Scope limits: give the model the narrowest permissions and tools that still get the job done, especially when it can take actions rather than write text.

The most important design decision is often what the system does when it doesn't know. A feature that admits uncertainty earns more trust than one that's fluent and wrong.

Cost and latency are product requirements

In a demo, nobody checks the bill. In production, tokens are a line item, and latency is a UX metric. Our first month's API spend was roughly triple our estimate, thanks to bloated context windows, retries on timeouts, and using the biggest model for tasks a smaller one handled just as well.

What moved the needle:

• Routing simple requests to smaller models and reserving larger ones for hard cases
• Caching repeated or near-identical queries
• Sending only relevant context instead of everything retrieved
• Streaming responses so perceived latency drops
• Setting budgets and alerts per feature, not just per account

A feature that's accurate but too expensive to run is still a failed feature.

Observability: you can't fix what you can't see

Traditional monitoring tells you whether a service is up. It doesn't tell you whether an answer was good. Log prompts, retrieved context, outputs, model versions, latency, and cost for every request, with sensible privacy handling. Add feedback signals like thumbs up and down, escalations, and follow-up questions that suggest what the first answer missed.

Then read a sample of real conversations every week. It's the fastest way to find failure patterns your eval set doesn't cover yet, and each one should flow back into the eval set. That loop of observe, add to evals, improve, redeploy is the real engine of quality.

Integration is where timelines go to die

A model call is one API request. The system around it is authentication, permissions, rate limits, queues, existing databases, and compliance. Small questions can turn into weeks. Which user's data can the model see? Where are conversations stored, and for how long? What happens during a provider outage?

This is also where teams decide whether to build everything themselves or bring in specialists. If your workflow is unusual, your data is sensitive, or the feature must plug into several internal systems, custom Generative AI development services can save months, because a team that has already built retrieval pipelines, eval harnesses, and guardrails won't have to learn those lessons on your budget.

For standard use cases, an off-the-shelf tool may be plenty. The honest answer depends on how specific your requirements are and how much of the 80% you want to own.

Scope before you build

The cheapest mistake to fix is the one made on paper. Many failed projects share a root cause: nobody defined success before the first line of code. Before building, answer these:

• What task or decision is this improving, and for whom?
• What does "good enough" mean in measurable terms?
• What does a wrong answer cost in this context?
• Is the data ready, and who owns it?
• Would something simpler, like rules, search, or a template, do the job?

This is where AI consulting earns its keep when done well. A good consultant's most valuable output is sometimes a smaller scope, a different approach, or a clear "not yet." Whether you bring in outside help or run it internally, the discipline matters more than who runs it.

The 80% in one checklist

1. Define success metrics before writing prompts.
2. Audit and clean the data the model relies on.
3. Build an eval set early and run it on every change.
4. Design for failure with validation, fallbacks, and human handoff.
5. Budget for tokens and latency from day one.
6. Log everything and review real conversations weekly.
7. Plan integration, permissions, and compliance up front.
8. Start with the narrowest useful scope, then expand.

Final thoughts

Prompting is the visible part of generative AI development, so it gets the attention. But the teams that ship reliable features are rarely the ones with the cleverest prompts. They treat the model as one component in a system that needs data quality, testing, safety nets, cost control, and monitoring.

The good news is that this is familiar territory. It's software engineering applied to a probabilistic component. If you've shipped production systems before, you already have most of the skills.

What does your 80% look like? Which part surprised your team most: evals, data, cost, or something I didn't cover? Tell me in the comments.

Frequently Asked Questions

1. What is generative AI development?

Generative AI development is the building of AI apps that create content, answer questions, and perform tasks using AI models, data, integrations, and evaluation systems.

2. Why is prompting only a small part of generative AI development?

Prompting guides the model's behavior, but production-ready AI also requires data preparation, evaluation, guardrails, cost optimization, monitoring, security, and integration with existing systems.

3. How do you improve the accuracy of a generative AI application?

Improve accuracy by cleaning the source data, using robust retrieval methods, building strong eval datasets, validating outputs, testing edge cases, and routing uncertain responses to a human.

4. What are the biggest challenges in deploying generative AI to production?

The key issues are unreliable outputs, poor data quality, prompt injection, high API costs, slow response times, privacy issues, and integration with existing business systems.

5. How can businesses make generative AI development more reliable?

Businesses can improve reliability by defining clear, measurable success criteria, testing every change against an evaluation dataset, implementing guardrails and fallbacks, monitoring real conversations, and starting with a very specific use case.