John Hu 實測評估將手寫 Kubernetes Agent 循環遷移至 OpenAI Agents SDK

A few weeks ago I wrote a small Kubernetes operations agent without using any agent framework. It is one Python for loop that talks to an LLM, calls cluster tools through MCP (Model Context Protocol), and writes a health report.

Then I read the example in the OpenAI Agents SDK repo. It wraps everything I had written by hand in a few clean abstractions: Agent, Runner, tools, tracing. The obvious question followed: should I just migrate?

I didn’t want to answer that from the docs alone, so I treated it as an engineering evaluation. I listed every behavior my agent has today, checked each one against the SDK source, and wrote a proof-of-concept script to measure the cases I couldn’t settle by reading.

Short answer: the migration is feasible, and it went better than my first estimate. But preserving my guardrails means spreading my loop logic across three different places. Along the way I found a real philosophical difference between my code and the SDK, and that difference is what this post is about.

All measurements below are against openai-agents 0.24.0, openai 3.35.1, mcp 1.30.0, and Python 3.13.

The loop I wrote by hand

The core of the agent fits on one screen:

for step in range(1, max_steps + 1):
    resp = await llm.chat.completions.create(
        model=model,
        messages=messages,
        tools=tools,
        tool_choice="required" if step == 1 else "auto",
    )

    msg = resp.choices[0].message
    messages.append(msg)

    if not msg.tool_calls:
        return msg.content  # after evidence checks

    for tc in msg.tool_calls:
        args = json.loads(tc.function.arguments)
        result = await mcp.call_tool(tc.function.name, args)
        messages.append({
            "role": "tool",
            "tool_call_id": tc.id,
            "content": result,
        })

That is the entire idea of an agent: ask the model, run the tools it asks for, feed the results back, repeat until it stops asking.

The interesting part is everything I had to add after the first version met a real model. Each guardrail exists because the model did something wrong at least once, and my git history reads like a list of failures:

1. It answered without investigating. So step 1 forces tool_choice="required", and if the model still answers without calling any tool, the loop pushes back once.

2. It called tools that don’t exist. My favorite was delete_pod. The loop replies with a tool message listing the tools that do exist.

3. It sent arguments of the wrong type. Every call is validated against the tool’s JSON Schema before it reaches the MCP server, and the model is told exactly which field to fix.

4. It called the same tool with the same arguments over and over. Duplicates return the cached result plus a nudge, and too many duplicates force a summary.

5. It invented pod names in the final report. Every pod name in the report must appear in some tool result; otherwise the loop pushes back once and names the invented pods.

6. It never converged. After MAX_STEPS, the loop removes the tools and asks for a summary of what it already has.

Each of these has a test. The tests use a scripted fake LLM, a tiny local HTTP server that plays a misbehaving model, against a real MCP mock server. Six tests, no API key needed.

Those six tests became the most important artifact in this whole evaluation.

Learning the SDK by mapping it onto my own code

The fastest way I found to learn a framework is to stop reading it as new concepts and read it as a set of names for things I already built.

Agent is my configuration. Instructions, tools, model, model settings. It doesn't run anything.

Runner is my for loop. Runner.run(agent, input, max_turns=10) calls the model, executes tool calls, appends results, and stops when the model produces a final answer. max_turns is my MAX_STEPS.

reset_tool_choice is my step-1 trick. This was the moment the SDK clicked for me. My loop forces tool_choice="required" on step 1 and relaxes it afterwards. The SDK has a flag for exactly that, and its docstring says it “ensures that the agent doesn't enter an infinite loop of tool usage.” Someone on the SDK team hit the same problem I did.

Tools are my MCP glue. @function_tool builds a JSON Schema from a Python signature. For lower-level control there is a raw FunctionTool, which receives the model's arguments as a JSON string. MCP servers can also be attached directly.

Tracing is my _save_trace, grown up. I save each run as a JSONL file of messages. The SDK records a tree of spans, one per LLM call, tool call, guardrail, and handoff, each with timing.

Guardrails and handoffs are things I hadn’t built yet. Output guardrails check the final answer. Handoffs let one agent transfer the conversation to another.

Having built the loop by hand first, almost every SDK parameter answered a question I had already asked myself. I don’t think I would have understood reset_tool_choice or tool_use_behavior nearly as quickly otherwise.

The method: tests as a migration contract

“Can I migrate?” is too vague to answer. “Can the SDK pass my six guardrail tests?” is not.

So the six tests became the contract, and every behavior in my agent got one of three verdicts: moves cleanly, moves with a redesign, or doesn’t move.

One practical detail before anything else worked: the SDK uses the OpenAI Responses API by default. My agent supports any OpenAI-compatible endpoint (DeepSeek, Qwen, Ollama), and my fake LLM speaks Chat Completions. The fix is to use the Chat Completions model class explicitly:

from openai import AsyncOpenAI
from agents import OpenAIChatCompletionsModel, set_tracing_disabled

set_tracing_disabled(True)  # otherwise traces try to upload to OpenAI

model = OpenAIChatCompletionsModel(
    model="fake-model",
    openai_client=AsyncOpenAI(
        api_key="test",
        base_url=fake_llm_url,
    ),
)

With that in place, I reused the same fake-server pattern from my tests and wrote a PoC script for the four behaviors I couldn’t settle by reading source.

What I measured

1. reset_tool_choice doesn’t send "auto". It sends nothing.

I expected request #2 to carry tool_choice: "auto". Here is what the fake server actually received:

request 1: tool_choice = "required"
request 2: tool_choice = <field absent>

The source resets tool_choice to None, and the Chat Completions converter omits a None value. For OpenAI that is equivalent to auto. But my test asserted the literal string "auto", so it would fail. The behavior matches; the assertion has to change.

Small, but this is exactly the kind of difference that only shows up when you check the request body.

2. Invalid JSON arguments: the decorator hides the error from the model

I made the fake model send {not-json as tool arguments, then tried both ways of defining a tool.

With @function_tool, my function never ran, and the model received this as the tool result:

An error occurred while running the tool. Please try again.

That’s a safe default for production, since it leaks nothing. But it’s useless for self-correction: the model has no idea what it did wrong.

With a raw FunctionTool, my handler received the raw string and could return a precise message, identical to what my hand-written loop sends:

Tool arguments are not valid JSON: Expecting property name enclosed
in double quotes: line 1 column 2 (char 1); raw arguments: {not-json

This decided the architecture: I would wrap my existing MCP client as raw FunctionTools instead of using the decorator or the native MCP integration.

3. Unknown tools: the default fails hard, and 0.24.0 has an escape hatch

When the fake model called delete_pod, the default behavior was:

ModelBehaviorError: Tool delete_pod not found in agent poc-unknown

The whole run aborts. Every AgentsException carries a run_data attribute with the run history. In my test the failure happened on the first turn, so there was nothing to recover, but even in principle catching the exception and rebuilding the conversation is clumsy.

My first draft of this evaluation filed unknown tools under “cannot migrate.” Then I read RunConfig more carefully and found this:

from agents import RunConfig

def whitelist(args):
    if args.kind == "tool_not_found":
        return (
            f"Unknown tool '{args.tool_name}'. "
            "Available tools: list_pods."
        )
    return None

run_config = RunConfig(
    tool_not_found_behavior="return_error_to_model",
    tool_error_formatter=whitelist,
)

With that config, the model received a normal tool output and the run continued:

Unknown tool 'delete_pod'. Available tools: list_pods.
Tool 'delete_pod' not found.

That is almost exactly my hand-written behavior.

Lesson learned: the examples show you the happy path; the config classes show you the edge cases.

4. Forced summaries: a sentinel works better than an exception

My loop has two ways to force a summary: too many duplicate calls, or too many steps. The SDK gives two candidate mechanisms.

max_turns raises MaxTurnsExceeded. Unlike the unknown-tool case, here run_data carried the full history in my test: four items (two tool calls, two tool outputs) and two raw model responses. So it is recoverable. I also noticed that 0.24.0 has run error handlers for the max_turns case, which I haven't tested yet.

tool_use_behavior turned out to be cleaner. It is a callback that runs after each round of tool calls and can declare the run finished:

from agents import ToolsToFinalOutputResult

FORCE_SUMMARY = "__FORCE_SUMMARY__"

async def stop_when_stuck(ctx, tool_results):
    if budget_exhausted():
        return ToolsToFinalOutputResult(
            is_final_output=True,
            final_output=FORCE_SUMMARY,
        )

    return ToolsToFinalOutputResult(is_final_output=False)

In my PoC, returning the sentinel ended the run with final_output == "__FORCE_SUMMARY__", and result.to_input_list() still contained the tool call and its output. From there, a forced summary is one more run of a tool-less clone of the agent.

The migration scorecard

Moves cleanly

• The core loop, which becomes Runner.run.
• Forcing tools on step 1, via tool_choice="required" plus the default reset_tool_choice (one test assertion changes).
• OpenAI-compatible providers, via OpenAIChatCompletionsModel.
• Logging, via RunHooks.
• Unknown tool feedback, via tool_not_found_behavior="return_error_to_model".

Moves with a redesign

JSON errors, schema validation, duplicate caching. These all move into a FunctionTool wrapper around my existing MCP client:

def wrap_mcp_tool(mcp, spec, validator, seen):
    async def invoke(ctx, raw_args: str) -> str:
        try:
            args = json.loads(raw_args or "{}")
        except json.JSONDecodeError as exc:
            return f"Tool arguments are not valid JSON: {exc}"

        errors = list(validator.iter_errors(args))
        if errors:
            return (
                f"Invalid arguments for {spec['name']}: "
                f"{errors[0].message}"
            )

        sig = f"{spec['name']}:{json.dumps(args, sort_keys=True)}"

        if sig in seen:
            return "[Duplicate call] Result unchanged:\n" + seen[sig]

        seen[sig] = await mcp.call_tool(spec["name"], args)
        return seen[sig]

    return FunctionTool(
        name=spec["name"],
        description=spec["description"],
        params_json_schema=spec["parameters"],
        on_invoke_tool=invoke,
        strict_json_schema=False,  # keep the MCP schema untouched
    )

Forced summaries. These move into tool_use_behavior plus a follow-up run.

Pushback on lazy answers and invented pods. These are the ones that change shape the most, and they deserve their own section.

Changes format

Traces. to_input_list() returns SDK items like function_call and function_call_output, not Chat Completions messages, so my existing JSONL replays would need a new format, or a custom trace processor.

The real difference: feedback vs. exceptions

Here is the pattern that kept coming up.

My hand-written loop treats almost every model mistake as feedback: tell the model what went wrong, append it to the conversation, let it try again. Wrong arguments, unknown tool, invented pod name — they’re all just more messages.

The SDK’s defaults treat model mistakes as exceptions: a ModelBehaviorError for unknown tools, a generic error string for bad arguments, a tripwire exception when an output guardrail fires. The run stops, and the developer decides what happens next.

Neither is wrong. They are designed for different failure models.

• Fail-fast is the right default for production systems. A model calling a tool that doesn’t exist might be a prompt injection, a version mismatch, or a bug, and you probably want to know rather than have the agent quietly improvise. Hiding exception details from the model also avoids leaking internals.
• Self-correction is the right choice when you expect the model to be wrong often and cheaply fixable, which is the everyday reality with smaller or open-weight models.

This difference shows up most clearly in my two pushback guardrails. In my loop, “you answered without investigating” and “these pods never appeared in any tool result” are just user messages appended mid-loop. Inside Runner, a plain-text answer is the final output; there is no hook to reject it and continue. Output guardrails can detect the problem, but they end the run with an exception rather than giving the model a second chance.

So pushback moves outside the Runner, into an outer loop:

result = await Runner.run(
    agent,
    question,
    max_turns=budget,
    run_config=run_config,
)

problem = check_report(result)  # no tools used? invented pods?

if problem:
    retry_input = result.to_input_list() + [
        {"role": "user", "content": problem}
    ]

    result = await Runner.run(
        agent,
        retry_input,
        max_turns=remaining,
        run_config=run_config,
    )

It works. But my single for loop has now become a tool wrapper, a tool_use_behavior callback, and an outer loop that has to split the step budget across multiple runs. Each piece is simple; the whole is harder to read in one sitting.

The SDK didn’t make my guardrails impossible. It made me decide, for each one, whether the mistake should be the model’s problem or the developer’s.

My decision

I’m keeping the hand-written loop as the main implementation of this project, because its whole point is to show how an agent works with nothing hidden. Next to it, I’m adding an SDK version built on the design above, run against the same six tests.

I would reach for the SDK first when:

• I need multiple agents with handoffs, where the SDK’s abstractions save real work.
• I need tracing I can inspect visually rather than grepping JSONL files.
• A team needs a shared vocabulary for agents instead of everyone’s own loop.
• The deployment is production-facing, where fail-fast defaults are a feature.

I would keep a hand-written loop when the guardrails are the product: when the value lies in exactly how the agent recovers from a bad model output.

Three takeaways

1. Build it by hand once before adopting the framework. Every SDK parameter I understood quickly was one I had needed myself. The abstractions stop being magic and become design decisions you can evaluate.

2. Turn “can we migrate?” into tests. My guardrail tests turned a vague judgment call into a list of pass, fail, and needs-redesign verdicts. They also caught a difference ("auto" vs. absent) that I would never have noticed by reading documentation.

3. Measure, and pin the version. My first draft said unknown-tool handling couldn’t be migrated. Version 0.24.0 proved me wrong. Agent frameworks are moving fast; any claim about their behavior has an expiry date.

Tested with openai-agents 0.24.0, openai 3.35.1, mcp 1.30.0, and Python 3.13.

The hand-written agent, its guardrail tests, and the PoC script (poc_sdk_behaviors.py) are on GitHub:

View the code: agent_loop_demo_for_kubernetes