如何使用 ASP.NET Core 建立 AI 應用程式

Introduction

Building an AI feature is easy if all you want is a textbox that sends a prompt to an LLM and prints the response.

Building an AI application that users can actually depend on is a different problem.

The moment an AI feature becomes part of a real application, you have to deal with authentication, conversation history, application data, failed model calls, prompt injection, rate limits, logging, testing, cost, and eventually scaling.

The LLM becomes one component inside your application rather than the application itself.

That is where ASP.NET Core fits well. It already gives .NET developers the HTTP pipeline, dependency injection, configuration, authentication, authorization, logging, middleware, testing ecosystem, and deployment options needed around an AI service.

In this article, we'll build a realistic AI Knowledge Assistant with ASP.NET Core and C#. We'll start with a simple model call, deliberately expose the weaknesses of that approach, and then evolve it into an application architecture that can handle context, private knowledge, security, failures, and production requirements.

What We Are Actually Building
Let's avoid building another imaginary /ask-ai endpoint that has no relationship with a real application. Assume we are building SupportDesk AI for a SaaS company.

The support team already has:
• Product documentation
• Troubleshooting guides
• Internal support notes
• Frequently asked questions
• API documentation
• Customer-support conversations

A support engineer should be able to ask:

"A customer is receiving a 401 response after enabling OAuth. What should I check first?

The application should answer using the company's approved knowledge rather than simply asking a general-purpose model to guess.
A simplified request looks like this:

The important design decision is that ASP.NET Core remains the application boundary.

The model does not become the owner of authentication, business rules, database access, or authorization.

The model generates language.
Our application decides what the model is allowed to see and what the user is allowed to do.

Start With a Small ASP.NET Core Application

For the first version, there is no reason to introduce microservices, Kubernetes, a vector database, an agent framework, and six different AI
providers.

Start with a normal ASP.NET Core Web API.
For example:

Then add the AI abstractions and the provider integration you intend to use.

Microsoft's current .NET AI documentation uses Microsoft.Extensions.AI as a common abstraction for AI functionality. The advantage is that application code can work against interfaces such as IChatClient instead of being tightly coupled to one model provider.

A simplified dependency setup can look like:

The exact provider packages depend on whether the application uses OpenAI, Azure OpenAI, a local model, or another compatible service.

Microsoft also documents local-model scenarios where .NET applications use Microsoft.Extensions.AI with a local provider such as Ollama.

That matters because the application architecture should not assume that every environment will use the same model.

2. The First Version: Put the AI Call Behind an Endpoint

The easiest implementation is also the one I would only keep for the first few minutes.

Suppose the API exposes:

With

The first implementation might look something like this:

There is nothing inherently wrong with this code.
In fact, this is a useful first test.

If the model responds, we have proven that:

1. ASP.NET Core can receive the request.
2. Dependency injection can provide the AI client.
3. The application can communicate with the model.
4. The model response can be returned through HTTP.

But this endpoint is not an AI application yet.
It is an AI-powered HTTP request.
Those are two different things.

3. The First Refactor: Keep AI Logic Out of the Controller

The second AI feature is usually where the first implementation starts becoming uncomfortable.

Soon the controller contains:
• System prompts
• Conversation history
• Model options
• Logging
• Retry handling
• Token limits
• Document retrieval
• Prompt construction
• Exception handling

At that point, the controller stops being an HTTP boundary and starts becoming an AI orchestration layer.

I would move that responsibility into an application service.
A simple structure could be:

The interface stays deliberately boring:

Then the implementation owns the AI-specific workflow:

Now the API layer becomes much smaller:

This may look like extra code for a simple endpoint.
It is.

And that is exactly the point.
The extra boundary becomes valuable when the application grows.

4. Configure the AI Client Through Dependency Injection

The application should not create an AI client inside every request.
Register it once and inject it where required.

For an Azure-hosted application, Microsoft documents using DefaultAzureCredential with Azure OpenAI and registering the resulting
IChatClient through dependency injection.

Conceptually:

The exact package versions and provider configuration should follow the current provider documentation rather than copying an old package list from a blog post.

The important architectural idea is more stable:

That separation makes future changes less painful.

5. Never Put AI Credentials in appsettings.json

This is one of those shortcuts that works perfectly on a developer laptop and becomes a security problem later.

Avoid:

The secret should not live in source control.
For local development, use the .NET user-secrets system:

For Azure-hosted applications, managed identity can remove the need to distribute long-lived credentials to the application. Microsoft
documents this pattern for ASP.NET Core applications accessing Azure OpenAI.

The production direction should therefore look more like:

The less secret material your application has to carry around, the less secret material you eventually have to rotate.

6. A Chatbot Needs Context, Not Just a Prompt

Consider these two requests:

The second question may depend entirely on the first conversation.
An application that sends only the latest message loses that relationship.
We therefore need conversation history.

A simple representation could be:

For a prototype, this history can live in memory.
For production, it usually belongs in persistent storage.

For example:

Now the request pipeline becomes:

Notice what is happening here.
The model is not responsible for remembering the conversation.
Our application is.

7. The Important Upgrade: RAG

Conversation history solves one problem.
It does not solve the knowledge problem.
Suppose the company changes its refund policy tomorrow.
The model does not automatically know that your company's internal policy changed.

This is where Retrieval-Augmented Generation, or RAG, becomes useful.
The basic flow is:

Microsoft's current .NET AI documentation includes RAG, embeddings, semantic search, vector stores, and data ingestion as core AI
application concepts.

For our SupportDesk application, the knowledge source might contain:

Instead of asking:

we build a grounded prompt:

The LLM is now operating with application-provided evidence.
That is a much more useful architecture than simply increasing the model temperature or writing a longer system prompt.

8. Don't Add a Vector Database Just Because It Is AI

This is an important engineering decision.
RAG does not automatically mean:

"Install a vector database."

If the knowledge base contains 200 small documents, a simple search implementation may be enough for the first version.

If the application has tens of thousands of documents and semantic retrieval becomes necessary, embeddings and vector search become
much more reasonable.

The architecture should evolve from the problem.
Not from the technology list.
A sensible progression is:

This approach also makes the system easier to debug.
When the AI gives a bad answer, you can ask:

Did retrieval return the wrong document?

rather than immediately blaming the model.

9. Treat Retrieved Content as Untrusted Input

There is another problem with RAG that is easy to miss.
A document retrieved from your knowledge base becomes part of the model's input.

That means the document itself can contain instructions.
Imagine a document contains:

If your system blindly gives that content to an agent with powerful tools, the consequences can be much worse than an incorrect answer.

OWASP identifies excessive agency as a major risk in LLM applications. The problem becomes especially important when models can invoke tools or interact with external systems. Their guidance emphasizes limiting functionality, permissions, and autonomy.

For a support assistant, I would initially keep permissions extremely narrow.

For example:

If business requirements eventually require actions, expose specific application functions rather than giving the model broad access.

For example:

is much safer as a tool boundary than giving the model unrestricted database access.

10. Authentication and Authorization Still Belong to ASP.NET Core

An AI endpoint should not be treated differently from any other sensitive API.

For example:

Then authorization should determine which knowledge the user can retrieve.

Imagine:

The RAG layer should enforce that boundary.
A common mistake is to retrieve everything first and then ask the LLM to "only use information the user is allowed to see." That is backwards.

Authorization should happen before the data reaches the model.
The model should never receive information that the requesting user was not permitted to access.

11. Handle AI Failures Like Normal Distributed-System Failures

An LLM call is a remote dependency.

It can:
• Timeout
• Return an error
• Become temporarily unavailable
• Hit a rate limit
• Return an incomplete response
• Become slower under load

So this:

cannot be the entire reliability strategy.
At minimum, introduce:

• Cancellation tokens
• Request timeouts
• Controlled retries
• Rate limiting
• Structured logging
• Meaningful error responses

For example:

I would not blindly retry every AI failure.
A timeout may be retryable.
An invalid request usually is not.
A rate-limit response may require backoff.
An authentication failure should not be retried five times.
Reliability logic needs to understand the type of failure.

12. Add Observability Before Production

A production AI application needs to answer questions such as:

Your logs should therefore contain useful metadata.

For example:

Be careful with the actual prompt and response content.

AI conversations can contain:
• Customer information
• Internal documents
• Credentials accidentally pasted by users
• Personal data

Logging everything because "we might need it later" is not a good observability strategy.

Log enough to diagnose the system without turning your logs into another sensitive data store.

13. Test the Application Without Calling the Model for Every Unit

Test

This is another place where AI projects can become expensive and difficult to maintain.

You don't want a unit test to depend on a live model.
Instead, abstract the AI dependency:

Then test your API behavior independently.

For example:

The live model belongs in a separate integration/evaluation layer.
That distinction is important:

A green unit-test suite does not prove that the AI answers are good.
Likewise, one successful chat session does not prove that your API is reliable.

They are different testing problems.

14. Prompt Construction Should Become Code, Not Random Strings

As soon as prompts become important, don't scatter them throughout controllers. Create a dedicated component.

For example:

Now prompt behavior can be tested.
It can also evolve independently from the HTTP layer.

This becomes increasingly important when you introduce:
• Structured output
• Tool calling
• Multiple assistants
• Different user roles

Microsoft's .NET AI documentation also includes structured output and function-calling scenarios as supported AI application patterns.

15. What I Would Not Build on Day One

This is probably the most useful part of the architecture discussion.
When building an AI application for the first time, it is tempting to add every interesting technology:

You can absolutely build that system.
But you don't necessarily need it.
For the first production-capable version of SupportDesk AI, I would start with:

Then introduce additional infrastructure when the application demonstrates the need for it.
This keeps the architecture understandable.
And when something breaks, there are fewer places to look.

16. The Final Architecture

After the first few iterations, our application has evolved from:

into a full application with a React client talking to an ASP.NET Core API, which coordinates Auth/AuthZ, a Conversation Service, and Rate
Limiting, all feeding into an Assistant Service. The Assistant Service in turn calls Knowledge Retrieval (backed by SQL / a vector store) and
an IChatClient (backed by the AI provider and, ultimately, the LLM).

This is still a normal web application.
The AI is an important dependency, but it has not taken over the architecture.

That distinction is what makes the application maintainable.

17. From Prototype to Production

A useful way to think about the development path is not "build the entire AI architecture."

Think in stages.

Stage 1 - Prove the model connection

Goal: Can the application communicate with the model?
Stage 2 - Add application boundaries

Goal: Can the AI logic evolve without turning controllers into a mess?
Stage 3 - Add real application context

Goal: Can the AI answer questions using application data?
Stage 4 - Add security and reliability

Goal: Can users safely depend on the application?
Stage 5 - Add evaluation and scaling

Goal: Can the application be operated and improved over time?

This is a much healthier development sequence than attempting to design a "perfect AI architecture" before the first user ever touches the
application.

Common Mistakes When Building AI Apps With ASP.NET Core

1- Putting the API key in source control. Secrets should be managed through appropriate configuration and identity mechanisms.
2- Calling the model directly from every controller. Create an application-level service boundary.
3- Assuming the LLM knows your business data. It doesn't automatically know your private documentation or current database state.
4- Adding RAG before understanding the problem. Retrieval is useful when the application actually needs external or private
knowledge.
5- Giving an AI agent excessive permissions. Limit tools, permissions, and autonomy. OWASP specifically highlights excessive agency
as a significant LLM application risk.
6- Testing only with live model calls. Separate application tests from AI evaluation.
7- Logging every prompt and response. AI conversations may contain sensitive information. Observability needs boundaries too.
8- Creating microservices too early. A well-structured ASP.NET Core application can take you surprisingly far before service
decomposition becomes necessary.

What ASP.NET Core Actually Contributes to an AI Application

It is easy to describe an AI application as "the LLM is the application." It isn't. The LLM is one component.

ASP.NET Core provides the surrounding engineering system: HTTP APIs, Authentication, Authorization, Dependency Injection, Configuration, Middleware, Validation, Logging, Caching, Rate Limiting, Health Checks, Testing, and Cloud Deployment.

The AI layer then adds: LLM, Embeddings, RAG, Structured Output, Function Calling, and AI Evaluation.

The real application is the combination of both.

That is why experienced .NET developers don't need to throw away their existing backend knowledge to enter AI development. They need
to learn where AI fits into a backend system.

Conclusion

Building an AI application with ASP.NET Core is not really about making one successful call to an LLM. That part is relatively straightforward. The harder engineering work starts afterward: protecting data, managing context, retrieving reliable knowledge, handling model failures, testing behavior, controlling permissions, and operating the system in production.

A practical approach is to start with a small ASP.NET Core API, isolate the AI dependency, then add conversation storage, RAG, security, observability, and evaluation as the application requires them. That keeps the system understandable while leaving room to scale.

If you're building a .NET application with AI capabilities, start with one real workflow rather than an oversized architecture. Prove the workflow, measure where it fails, and let those failures drive the next architectural decision.

Recommended Official References

Microsoft Learn - AI apps for .NET developers - current .NET AI learning hub covering LLMs, RAG, embeddings, agents, structured
output and related tooling.

Microsoft Learn - Build an AI chat app with .NET - demonstrates Microsoft.Extensions.AI with OpenAI/Azure OpenAI.

Microsoft Learn - Develop .NET apps with AI features - overview of the current .NET AI ecosystem and application patterns.

Microsoft Learn - Authenticate an Azure-hosted .NET app to Azure OpenAI - managed identity and Microsoft Entra ID authentication
pattern.

OWASP GenAI Security Project - Excessive Agency - security guidance for limiting LLM functionality, permissions and autonomy.

More practical AI & .NET engineering guides:
AI Engineering Foundry - https://aiengfoundry.com