如何以 OpenTelemetry、Microsoft Agent Framework 和 Azure Monitor追蹤多智能體 AI 系統

This blog post is part of the Agentic Azure Insights community initiative created by Nicholas Chang. Take a look at other sessions that are part of the event.

Building, tracing, monitoring, and troubleshooting intelligent multi-agent applications with Python, Microsoft Foundry, and Azure.

Introduction

Building an AI agent that responds to a prompt is relatively straightforward. Building a production-ready multi-agent application is a completely different challenge.

As AI systems evolve from individual agents into collaborative architectures, understanding their behavior becomes increasingly difficult.

Imagine an application where several specialized AI agents collaborate to solve a customer support request. One agent retrieves customer information, another evaluates business policies, and a third generates the final resolution.

What happens when the application becomes slow? How do we identify which agent caused the delay? What if all agents execute successfully, but the final decision is incorrect?

Traditional application logs provide some answers, but they are often insufficient for reconstructing complex, distributed AI workflows.

This is where OpenTelemetry, Microsoft Agent Framework, and Azure Monitor become valuable.

In this tutorial, we'll build an observable multi-agent application that demonstrates how to:

• Instrument distributed agent workflows using OpenTelemetry.
• Track interactions between agents and external tools.
• Correlate operations using trace IDs and spans.
• Export telemetry to Azure Monitor and Application Insights.
• Investigate performance bottlenecks and unexpected decisions.
• Use Kusto Query Language (KQL) to analyze application behavior.
• Deploy the application to Azure Container Apps with an automatically provisioned Microsoft Foundry project and gpt-4o-mini model.

Most importantly, we'll investigate actual scenarios where something goes wrong.

Technologies that will be used in the tutorial: Python, FastAPI, Microsoft Agent Framework, Microsoft Foundry, OpenTelemetry, Azure Monitor, Application Insights, KQL, Azure Container Apps, Bicep, and Azure Developer CLI.

1. Understanding Observability in Multi-Agent Systems

Observability is the ability to understand the internal behavior of a system by analyzing the signals it produces.

In traditional distributed applications, observability relies on three fundamental pillars:

For agentic AI applications, these capabilities become even more important.

A multi-agent workflow might involve multiple model invocations, external APIs, database queries, and decisions based on information produced by other agents.

A single user request can generate dozens of operations across different components.

Without distributed tracing, reconstructing the execution path becomes difficult.

OpenTelemetry and distributed tracing

OpenTelemetry provides a vendor-neutral framework for generating, collecting, and exporting telemetry.

Its tracing model is particularly useful for multi-agent systems.

A trace represents an end-to-end operation, while a span represents an individual unit of work within that operation.

For example, consider the following execution:

Every span contains information such as its operation name, execution time, status, attributes, and relationship with other spans.

All spans within the same distributed operation share a trace ID.

This allows us to reconstruct how the application produced a particular result.

2. Our Demo: A Multi-Agent Customer Support Investigator

Let's build a small multi-agent application that investigates customer refund requests.

The architecture includes a coordinator and three specialized agents.

CustomerAgent analyzes customer information, including account tier and purchase history.

PolicyAgent examines refund policy evidence and determines whether the available information supports a refund.

ResolutionAgent combines the findings and generates a recommended resolution.

The coordinator handles the execution sequence, invokes tools, and supplies the evidence required by each agent.

Our application also includes two simulated external services: a customer information API and a refund policy lookup API.

Application architecture

The application supports two execution modes.

In mock mode, agent responses are deterministic, making it possible to demonstrate observability without a deployed AI model.

In Foundry mode, the application creates real Microsoft Agent Framework agents connected to the Microsoft Foundry project and model deployment that are created as part of the Azure deployment. In the default template, the project points to a provisioned gpt-4o-mini deployment so the app can run with real model inference without requiring separate manual setup.

This distinction is important because the mock mode demonstrates workflow instrumentation, while Foundry mode adds real model execution and associated telemetry when supported by the installed SDK.

3. Creating the Python Project

Our repository is organized as follows:

GitHub repository is available here.

The main dependencies include:

fastapi
uvicorn
opentelemetry-api
opentelemetry-sdk
opentelemetry-instrumentation-fastapi
azure-monitor-opentelemetry
azure-identity
agent-framework-core
agent-framework-foundry
python-dotenv
jinja2

We can create the environment and install the dependencies:

python3 -m venv .venv
source .venv/bin/activate

pip install -r requirements.txt

cp .env.example .env

Initially, we'll use the following configuration:

MODEL_MODE=mock
APPLICATIONINSIGHTS_CONNECTION_STRING=
OTEL_SERVICE_NAME=multi-agent-investigator
OTEL_CONSOLE=true
ENABLE_SENSITIVE_DATA=false

This allows the application to operate locally without any Azure resources.

4. Configuring OpenTelemetry

Let's begin with the most important technical component: configuring the OpenTelemetry SDK.

In app/telemetry.py, our implementation supports both local console tracing and Azure Monitor.

First, we import the necessary components:

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    BatchSpanProcessor,
    ConsoleSpanExporter
)

Next, we configure a tracing provider for local execution:

provider = TracerProvider(
    resource=Resource.create({
        "service.name": "multi-agent-investigator"
    })
)

provider.add_span_processor(
    BatchSpanProcessor(ConsoleSpanExporter())
)

trace.set_tracer_provider(provider)

The TracerProvider manages tracing configuration, while the BatchSpanProcessor processes spans and sends them to the configured exporter.

The ConsoleSpanExporter prints the generated spans to the terminal.

This is especially useful during local development and live demonstrations.

For Azure Monitor, the project uses a different configuration path:

from azure.monitor.opentelemetry import configure_azure_monitor

configure_azure_monitor(
    connection_string=connection,
    resource=Resource.create({
        "service.name": service
    }),
    sampling_ratio=1.0,
    logger_name="agent_demo",
    enable_live_metrics=True
)

The application selects this configuration when APPLICATIONINSIGHTS_CONNECTION_STRING is present.

It can also attach the console exporter to the same tracing provider, allowing us to observe telemetry locally while exporting it to Azure.

We deliberately configure only one global tracing provider.

Enabling Microsoft Agent Framework instrumentation

Now comes another important integration:

from agent_framework.observability import enable_instrumentation

enable_instrumentation(
    enable_sensitive_data=False
)

Microsoft Agent Framework can produce OpenTelemetry signals for instrumented agent operations.

This complements the custom spans we create for our coordinator, workflow, and simulated external tools.

The enable_sensitive_data=False setting is particularly important because agent inputs and outputs may contain confidential or personally identifiable information.

In production environments, capturing entire prompts, customer records, and model responses should never be enabled without appropriate privacy controls.

5. Creating the Specialized Agents

Our next step is creating the agents.

In app/agents.py, we define the instructions for each specialist:

AGENT_INSTRUCTIONS = {
    "CustomerAgent": (
        "You are a customer record specialist. "
        "Summarize the provided account facts "
        "in one sentence. Do not invent additional facts."
    ),

    "PolicyAgent": (
        "You review refund policy evidence. "
        "Report whether the supplied policy "
        "evidence allows refund."
    ),

    "ResolutionAgent": (
        "Provide a concise support resolution "
        "based ONLY on the supplied customer "
        "and policy evidence."
    )
}

When Foundry mode is enabled, the application initializes a Foundry client:

from agent_framework import Agent
from agent_framework.foundry import FoundryChatClient
from azure.identity import DefaultAzureCredential

client = FoundryChatClient(
    project_endpoint=endpoint,
    model=model,
    credential=DefaultAzureCredential()
)

Then we create the specialized agents:

agents = {
    name: Agent(
        client=client,
        name=name,
        instructions=instructions
    )
    for name, instructions in AGENT_INSTRUCTIONS.items()
}

Each agent is responsible for a particular task.

The coordinator supplies the evidence and invokes the agents sequentially.

For example:

result = await agents["PolicyAgent"].run(
    policy_evidence
)

Notice that our orchestration does not require the agents to communicate directly with each other.

The Python coordinator explicitly controls execution and information exchange.

This makes the workflow easier to understand and allows us to create custom tracing boundaries around every operation.

6. Instrumenting the Multi-Agent Workflow

Now let's connect the pieces.

In app/workflow.py, we create a root span to represent the entire investigation.

with tracer().start_as_current_span(
    "workflow.investigate",
    attributes={
        "demo.scenario": scenario,
        "demo.request_id": request_id,
        "workflow.name": "customer_investigation"
    }
) as root:

    ctx = root.get_span_context()

    trace_id = f"{ctx.trace_id:032x}"

This is the starting point of our observable workflow.

Every child span created within this context inherits the current tracing context.

The code also extracts the trace ID and returns it to the web interface.

This gives the user a direct way to correlate an application response with its telemetry.

Next, we instrument the coordinator:

with tracer().start_as_current_span(
    "agent.Coordinator",
    attributes={
        "agent.name": "Coordinator",
        "demo.scenario": scenario
    }
):
    # Execute the workflow
    ...

We then create individual spans for the specialized agents:

with tracer().start_as_current_span(
    "agent.CustomerAgent",
    attributes={"agent.name": "CustomerAgent"}
):
    customer_out = await runner.run(
        "CustomerAgent",
        customer_evidence
    )

The same pattern is applied to PolicyAgent and ResolutionAgent.

This allows us to visualize the workflow hierarchy and distinguish agent operations from external tool calls.

Importance of custom attributes matter

OpenTelemetry spans support custom attributes that add context to an operation.

In our application, we use attributes such as:

demo.scenario
demo.request_id
agent.name
tool.name
demo.outcome
demo.policy_version
demo.answer_quality

These attributes are particularly useful when analyzing telemetry in Azure Monitor.

Instead of searching through unstructured log messages, we can filter operations by their business meaning.

For example, we can investigate only the executions where the policy version was stale or where the workflow produced an unexpected outcome.

7. Building the Web Interface

To make the demo interactive, we use FastAPI and Bootstrap.

The web application exposes an endpoint to execute a selected scenario:

@app.post("/api/scenarios/{scenario}")
async def run_scenario(scenario: str):

    if scenario not in VALID_SCENARIOS:
        raise HTTPException(
            status_code=404,
            detail="Unknown scenario"
        )

    return await investigate(
        scenario,
        app.state.runner
    )

The full project also provides a health endpoint and an HTML interface.

The user can select four scenarios:

1. Healthy execution.
2. Slow external API.
3. Incorrect decision caused by stale policy information.
4. Upstream tool failure.

After executing a scenario, the web application displays the final recommendation, intermediate agent outputs, execution duration, request ID, and trace ID.

The trace ID is particularly valuable because we can copy it directly into our KQL queries.

Let's examine each scenario.

8. Demo 1: Understanding a Healthy Execution

First, let's execute the healthy scenario.

We can use the web interface or invoke the API:

curl -X POST \
  http://localhost:9001/api/scenarios/healthy

The coordinator performs the following operations:

1. Retrieves customer information.
2. Sends that information to CustomerAgent.
3. Retrieves the refund policy.
4. Sends the policy to PolicyAgent.
5. Combines the evidence using ResolutionAgent.

The simulated customer is a premium customer whose purchase occurred 12 days ago.

The refund policy allows premium customers to request a refund within 30 days.

Consequently, the expected recommendation is to approve the refund.

What should we observe?

The console exporter generates spans for the workflow, coordinator, tools, and agents.

These spans share the same trace ID and contain timing information.

When using Foundry mode, supported automatic agent instrumentation adds model execution details.

The key observation is that we now understand much more than the final answer.

We can reconstruct the execution sequence and see where time was spent.

This is the baseline against which we will compare our next scenarios.

9. Demo 2: Investigating a Slow API

Imagine that customers are reporting slow responses from our AI application.

The first assumption might be that the model is taking too long to respond.

However, the actual problem could be an external service.

To demonstrate this, we inject latency into the customer lookup operation.

async def _customer_tool(scenario: str):

    with tracer().start_as_current_span(
        "tool.get_customer",
        attributes={
            "tool.name": "get_customer",
            "demo.scenario": scenario
        }
    ):

        if scenario == "slow_api":
            await asyncio.sleep(2.5)
        else:
            await asyncio.sleep(0.08)

        return {
            "customer_id": "C-1001",
            "tier": "premium",
            "order_id": "O-2048",
            "days_since_order": 12
        }

Execute the scenario:

curl -X POST \
  http://localhost:9001/api/scenarios/slow_api

The workflow still produces the expected answer, but its total duration increases.

The trace makes it possible to identify the source of this latency.

Rather than blaming the model, we can see that tool.get_customer consumed approximately 2.5 seconds.

This is an important distinction.

Performance problems in agentic applications are not necessarily model problems.

They can originate from databases, tool servers, network calls, authentication services, or other dependencies.

Investigating with KQL

Once telemetry reaches Application Insights, we can identify the slowest operations:

dependencies
| where timestamp > ago(24h)
| where name has "tool."
    or name has "agent."
    or name has "workflow."
| summarize
    runs = count(),
    p50_ms = percentile(duration / 1ms, 50),
    p95_ms = percentile(duration / 1ms, 95),
    failures = countif(success == false)
    by name
| order by p95_ms desc

This query measures operation latency and failure counts.

With repeated executions, percentiles such as P95 help reveal performance patterns that averages can hide.

For this demonstration, we expect the customer lookup operation to exhibit significantly higher latency during the slow API scenario.

10. Demo 3: When an Agent Makes the Wrong Decision

This is probably the most interesting scenario in the demonstration.

What happens when all the components execute successfully, but the final answer is incorrect?

Distributed tracing can tell us whether an operation completed, but it does not automatically establish whether the answer is factually correct.

Consider a refund policy service that returns outdated information.

Our application simulates this behavior:

if scenario == "incorrect_decision":

    span.set_attribute(
        "demo.fault",
        "stale_policy_data"
    )

    return (
        "REFUND_DENIED: stale cached policy "
        "says all refunds are disabled"
    )

The customer is eligible for a refund according to the current policy, but the tool returns an outdated rule.

In deterministic mock mode, PolicyAgent follows that information and ResolutionAgent recommends declining the refund and requesting manual review.

Technically, the workflow completes without exceptions.

However, the decision is incorrect.

Recording the quality signal

To make this problem observable, our demo attaches custom attributes to the root span:

incorrect = scenario == "incorrect_decision"

root.set_attribute(
    "demo.answer_quality",
    "incorrect_due_to_stale_tool"
    if incorrect else "expected"
)

root.set_attribute(
    "demo.policy_version",
    "stale" if incorrect else "current"
)

root.set_attribute(
    "demo.outcome",
    "incorrect" if incorrect else "success"
)

This allows us to distinguish technical success from expected business behavior.

There is an important limitation: these labels are generated by the demonstration's known fault-injection scenario.

They do not represent an automatic AI evaluation.

A production system would need additional validation, policy freshness checks, rule-based verification, or model-based evaluation to detect unexpected incorrect decisions.

In Foundry mode, the model may even recognize the conflict and avoid the incorrect recommendation.

Investigating quality signals using KQL

dependencies
| where timestamp > ago(24h)
| where name has "workflow.investigate"
| extend
    outcome = tostring(
        customDimensions["demo.outcome"]
    ),
    scenario = tostring(
        customDimensions["demo.scenario"]
    ),
    policyVersion = tostring(
        customDimensions["demo.policy_version"]
    )
| summarize
    runs = count(),
    avg_ms = avg(duration / 1ms)
    by scenario, outcome, policyVersion

This query helps identify executions with intentionally labeled incorrect outcomes.

The broader lesson is that AI observability requires combining operational telemetry with application-level quality signals.

A successful HTTP request or completed span does not guarantee a trustworthy AI decision.

11. Demo 4: Troubleshooting an External Tool Failure

Our final scenario simulates a failure in the policy service.

Instead of returning a policy, the tool raises an exception.

if scenario == "tool_error":

    span.set_attribute(
        "demo.fault",
        "upstream_timeout"
    )

    raise TimeoutError(
        "Policy service timed out "
        "(intentional demo fault)"
    )

At the workflow level, we capture the exception and mark the root span as failed:

except Exception as exc:

    root.record_exception(exc)

    root.set_status(
        Status(StatusCode.ERROR, str(exc))
    )

    root.set_attribute(
        "demo.outcome",
        "error"
    )

This gives us diagnostic information about the failure, including its relationship with the workflow.

An interesting detail is that the demo returns a structured JSON response describing the failed investigation.

This means the web endpoint itself may still return HTTP 200.

Consequently, simply monitoring HTTP status codes would not provide a complete picture of application failures.

Explicit workflow-level error signals are essential.

Finding errors using KQL

union isfuzzy=true dependencies, exceptions
| where timestamp > ago(24h)
| where success == false
    or itemType == "exception"
| project
    timestamp,
    itemType,
    operation_Id,
    name,
    type,
    outerMessage,
    customDimensions
| order by timestamp desc

This allows us to investigate failed dependencies and exception telemetry.

The trace ID connects the problem to the corresponding customer investigation.

12. Exporting Telemetry to Azure Monitor

So far, we have been examining telemetry locally.

Now let's centralize the information in Azure.

Our Bicep infrastructure defines several resources:

• Azure Log Analytics Workspace.
• Azure Application Insights.
• Azure Container Apps Environment.
• Azure Container Registry.
• Azure Container App.
• Azure AI Foundry / Azure AI Services account.
• Foundry project.
• Model deployment for gpt-4o-mini.

Application Insights is connected to the Log Analytics workspace, and the app receives the provisioned Foundry endpoint and model deployment settings as environment variables.

The Container App receives an Application Insights connection string through its environment configuration.

For example, the Bicep template includes:

resource insights 'Microsoft.Insights/components@2020-02-02' = {
  name: appInsightsName
  location: location
  kind: 'web'
  properties: {
    Application_Type: 'web'
    WorkspaceResourceId: logs.id
  }
}

The deployment configuration is defined in azure.yaml:

name: multi-agent-observability-lab

infra:
  provider: bicep
  path: infra

services:
  web:
    project: .
    language: py
    host: containerapp
    docker:
      path: Dockerfile

We can provision and deploy the infrastructure using Azure Developer CLI:

az login
azd auth login

azd env new agent-otel-demo
azd env set AZURE_LOCATION eastus2

azd up

This workflow is designed to provision the hosting and observability resources, build the container image, and deploy the application.

The Bicep template uses a placeholder image during initial infrastructure provisioning, which the application deployment is intended to replace.

A couple of considerations are worth mentioning.

First, the Azure deployment provisions the Microsoft Foundry account, project, and gpt-4o-mini model automatically as part of the same infrastructure stack, so the sample can run with live inference without a separately managed Foundry setup.

Second, the Azure deployment template has not been end-to-end validated against a live subscription as part of this tutorial. Azure permissions, regional availability, SDK changes, and container deployment behavior should be tested before using the demo in a live event.

For a reliable presentation, we can also keep the Python application running locally while exporting telemetry to a provisioned Application Insights resource.

This avoids making the entire demonstration dependent on a successful cloud application deployment.

13. Correlating All Operations Using a Trace ID

One of the most useful observability features is the ability to reconstruct a complete execution using a single identifier.

Our web interface displays the trace ID returned by the workflow.

We can copy that identifier and execute the following query in Application Insights:

let traceId = "REPLACE_WITH_TRACE_ID";

union isfuzzy=true
    requests,
    dependencies,
    traces,
    exceptions

| where operation_Id == traceId

| project
    timestamp,
    itemType,
    name,
    duration,
    success,
    message,
    operation_Id,
    parentId = operation_ParentId,
    customDimensions

| order by timestamp asc

This brings together the telemetry associated with a single investigation.

Because the operations share the same correlation context, we can identify the events, spans, and failures that belong to the same workflow.

This is especially useful as multi-agent architectures grow beyond one application.

For example, the same principles can support correlation across HTTP services, agent tools, databases, and other distributed components, provided trace context is propagated across those boundaries.

14. Observing Model Token Consumption

Another important aspect of AI observability is cost.

With traditional applications, we typically analyze infrastructure usage, CPU, memory, and service request counts.

Generative AI introduces additional consumption indicators, including input tokens, output tokens, and model invocations.

Microsoft Agent Framework instrumentation can expose model usage information when supported by the specific SDK version, model provider, and exporter.

Our project includes a query that looks for common generative AI telemetry attributes:

dependencies
| where timestamp > ago(24h)
| extend
    inputTokens = tolong(
        customDimensions["gen_ai.usage.input_tokens"]
    ),
    outputTokens = tolong(
        customDimensions["gen_ai.usage.output_tokens"]
    )
| where
    isnotnull(inputTokens)
    or isnotnull(outputTokens)
| summarize
    input_tokens = sum(inputTokens),
    output_tokens = sum(outputTokens),
    calls = count()
    by name
| order by input_tokens desc

This query is relevant when running real model inference through Foundry.

In mock mode, no model calls are executed, so token usage is not generated.

Additionally, the exact telemetry attribute names and storage locations may vary depending on instrumentation versions and Azure Monitor mappings.

Before building production dashboards, we should inspect actual exported model spans and validate the available attributes.

Once reliable usage data is available, we can calculate estimated model costs by combining token consumption with the pricing of the selected model deployment.

This creates opportunities for agent-level cost allocation, budget monitoring, and operational optimization.

15. What About Production Environments?

Our demonstration focuses on understanding observability concepts, but production-ready multi-agent applications require additional considerations.

Distributed context propagation: When agents and tools execute across multiple services or processes, tracing context must be propagated between them. Creating local child spans is only the starting point.

Sampling and telemetry volume: Our demo records all traces to make investigation easier. Production applications should balance diagnostic requirements, sampling decisions, ingestion costs, and retention policies.

Security and privacy: Telemetry must not indiscriminately capture user prompts, confidential documents, credentials, or personal information.

Automated anomaly detection: KQL queries and Azure Monitor alert rules can identify patterns such as increased failure rates, abnormal latency, or unexpected usage.

AI evaluation: Operational tracing should be complemented by evaluation techniques that assess accuracy, groundedness, policy compliance, and other application-specific quality dimensions.

Cost governance: Token usage, model calls, infrastructure consumption, and telemetry ingestion should be incorporated into the application's non-functional requirements and operational budgets.

Observability is not simply a debugging feature. It is part of operating AI systems responsibly at scale.

16. Running the Entire Demo

Let's summarize the recommended execution sequence.

First, start the application locally:

python -m uvicorn app.main:app --reload

Open:

http://localhost:9001

Then execute the scenarios in this order:

For each scenario, examine the console traces, copy the trace ID, and run the corresponding KQL queries in Azure Monitor.

If Application Insights telemetry is not immediately visible, use the local console exporter while ingestion completes.

Finally, switch to Foundry mode to demonstrate real agent execution and model-related telemetry.

The default Azure deployment already creates the project endpoint and gpt-4o-mini model, so the app can be pointed to the provisioned values without manual Azure AI setup. If you want to override them manually, the environment variables are:

MODEL_MODE=foundry
FOUNDRY_PROJECT_ENDPOINT=https://YOUR-PROJECT.services.ai.azure.com
FOUNDRY_MODEL_DEPLOYMENT=YOUR_MODEL_DEPLOYMENT

Authenticate with Azure and ensure that your identity has the necessary permissions to invoke the deployed model.

Restart the application and execute the same scenarios.

Now, instead of deterministic mock responses, the agents use the Microsoft Foundry deployment provisioned by the infrastructure.

Conclusion

Multi-agent AI applications introduce new operational challenges because their behavior depends on multiple agents, models, tools, and external services.

A final response tells us what the application produced, but rarely explains the complete execution path that led to it.

OpenTelemetry provides the tracing foundation needed to understand that execution path, while Microsoft Agent Framework offers instrumentation for agent operations and Azure Monitor enables centralized analysis.

Through our four demo scenarios, we learned how to identify a performance bottleneck, correlate failures, investigate incorrect decisions, and explore model usage.

Perhaps the most important lesson is that observability and evaluation solve complementary problems.

Observability helps us understand what happened inside the system. Evaluation helps us determine whether the behavior and outcomes were appropriate.

Combining both capabilities is essential for building reliable, explainable, and operationally manageable agentic AI systems.

Resources

• Microsoft Agent Framework documentation
• Microsoft Agent Framework observability
• OpenTelemetry documentation
• Azure Monitor OpenTelemetry overview
• Azure Monitor Logs and KQL
• Azure Developer CLI documentation
• Microsoft Foundry

Don't forget to check the rest of Agentic Azure Insights community sessions.

Thank you for reading, see you next time!
Luis