OpenAI Agents SDK 一度看似流失近半引用，多數其實只是格式不同

Five pre-registered experiments, one at a time, the moment reading the raw answers changed the conclusion, the re-score, and the three runs per setup that finally told me what was actually true.

Part 2 of 6 in the series Citations at the MCP Boundary. New here? Start with Part 1: Your MCP Server Returns the Source. Does the Answer Still Cite It?. The full list of parts is at the end of this post.

In Part 1 I described a benchmark for whether citations survive when an AI agent consumes a RAG knowledge base over MCP. On the first pass, the headline was uncomfortable for one framework:

A gap that size between two agents running the same model and the same prompt doesn't happen by accident. Something specific was going on, and until I knew what, the result was a number rather than a finding.

This post is the investigation. It's the part of the project I'm proudest of, because the interesting answer only appeared after the obvious ones had been ruled out properly.

Step 0: is it even real?

My first instinct was that the result was a bug in my code, not a property of the framework.
0.323 vs 0.614 on identical inputs smells like a parsing or logging problem.

So before any theory, I checked the boring explanations:

• Was the trace logging broken? No. Every SDK answer had a non-empty retrieval trace. The agent did search, and it did get results.
• Were the prompts different? No. The two agent prompts are byte-identical.
• Was the scorer misreading answers? The scorer looks for citations in the required [url] format. 66 of the 127 SDK answers had no citation in that format, in otherwise well-formed prose.

So the effect was real at the level of the outputs. Now: why?

Writing the plan down first

There were several plausible causes, and I had a favourite. That's exactly when it's easiest to fool yourself: you keep running variations until one confirms the idea you already had.

So before running anything, I wrote the plan into the project's decision log:

• One arm: OpenAI Agents SDK × gpt-4o-mini, all 127 questions, same eval set, same prompt, same scorer, same citation format.
• One change per run, each compared against the run before it.
• One rule for "this helped": the paired bootstrap 95% confidence interval of the per-question CSR difference (10,000 resamples, fixed seed) must exclude zero.
• Every result gets reported, whichever way it comes out.

Paired is the important word there. Every run answers the same 127 questions, so the right comparison is question by question, which removes most of the noise from questions simply being easier or harder.

Hypothesis 1: the agent loop

This was my favourite. Agents search, read, maybe search again, then write. By the time the final answer is written, the sources from the first tool call are many messages back. Surely they get lost along the way?

It didn't survive contact with the data I already had:

• LangGraph runs the same kind of loop and lost nothing. Its difference from single-shot RAG, question by question, was +0.000.
• The agents barely looped. The SDK agent made about 1.3 searches per answer, LangGraph about 1.2. There wasn't much of a "long multi-step context" for sources to get lost in.

So the loop wasn't it. That left the question of what was different between two agents doing nearly the same thing.

Hypothesis 2: temperature (a confound I found in my own code)

While writing the plan I read both pipelines line by line, and found something I'd missed: the SDK pipeline never set a temperature. The SDK's model settings leave it unset, so the API default was used. The LangGraph pipeline ran at temperature 0.

So "same model, same prompt" was true, but "same settings" wasn't. A more random sampler could plausibly be sloppier about formatting. This had to be tested before anything else, because if it
explained the gap, the headline was an artifact of my setup.

Two useful results. First, the headline replicated: a fresh run of the original setup scored 0.333 against the published 0.323. Second, matching the temperature helped a little, but not significantly, and E1 was still clearly below single-shot RAG (−0.207, a real difference).

Temperature was a genuine flaw in my setup, and fixing it was right. But it wasn't the cause.

Hypothesis 3: the tool description

The MCP tool comes with a description the model reads. A real hosted MCP retrieval product publishes recommended wording for it, so I tested that next (E2): same as E1, with the recommended tool description served by the MCP server.

Nothing. That fits the product's own documentation once I read it at the source: it puts the citation rules in the system prompt, and the recommended tool description only says that chunks carry a source URL. (An earlier summary of a blog post had told me the opposite. Lesson: read the primary source.)

Hypothesis 4: the API underneath

This is where it got interesting. The OpenAI Agents SDK talks to OpenAI's Responses API by default. LangGraph's OpenAI integration uses the older Chat Completions API. Same model name, different API, and potentially a different way of delivering instructions and tool outputs to the model.

Before running the experiment I wanted to rule out the transport itself. Maybe the SDK was reformatting tool results and dropping the URLs before the model ever saw them? I added a flag to dump exactly what the model receives. (It only got through one question before my OpenAI credits ran out, which is its own lesson.) On that question, the system prompt arrived intact, and the tool output arrived as one block of JSON, about 12,000 characters, with every chunk's source_url intact. The SDK wasn't hiding the sources.

Then the experiment (E3): same as E1, but the SDK agent uses Chat Completions.

That looked like the one. Answers without a [url] citation dropped from 54 to 18, the same level as single-shot RAG. (Hold on to the words "without a [url] citation". They turn out to matter.)

Hypothesis 5: following the published guidance in full

One more arm, added (and written down) before running it: what if a developer follows the retrieval provider's published guidance completely, meaning their recommended system prompt plus tool description?

A real improvement by this measure, and enough that the SDK was no longer significantly below single-shot RAG. One caveat: that prompt asks for a different citation format ([[title](url)]), so this run changes the wording and the format together and can't separate them.

The twist: reading the "uncited" answers

At this point I had a clean story: the default API loses citations, and switching APIs (or using a good system prompt) recovers them.

Then I did what I should have done much earlier, and read the 66 "uncited" answers from the published run one by one.

Most of them weren't uncited. They looked like this:

...the response model used for returning user data (like UserOut) specifically does not
include the password or any hashed password fields. source_url

The right page, right after the claim, just written as a Markdown link instead of the required [url]. Counting them up:

For comparison, single-shot RAG has 19 answers with no URL anywhere. On the crude question "does the answer mention any URL at all", the SDK agent was no worse than the baseline. (Whether it cited the right sources as often is a separate question, and it took three runs to answer it; see below.)

And the same breakdown across the experiments shows what the API switch actually fixed:

The answers with no URL at all barely changed (11 → 9). What collapsed was the Markdown links (41 → 9). Chat Completions didn't make the model cite more. It made the model follow the citation
format it was asked for.

So the accurate version of the finding is this:

With the same model and prompt, the OpenAI Agents SDK's default (the Responses API) ignored the
required citation format in 51 of 127 answers (40%), mostly writing Markdown links instead.
On Chat Completions that falls to 9, close to LangGraph (8) and single-shot RAG (0).

Is that less important than "the agent forgot its sources"? Not for anyone building a product.
Every system that consumes citations does it by format: the "view sources" panel, the click-through tracking, the guardrail, the eval harness. When the format breaks, the citation is gone as far as the system is concerned. And unlike "the agent forgot", it's specific, testable and fixable.

But it also meant every CSR number above was measuring two things at once: whether the agent cited the right source, and whether it used the exact format my parser recognised. I had to separate
them.

You can reproduce the breakdown yourself from the committed outputs:

import json, re

CITE = re.compile(r'\[(https?://[^\]\s]+)\]')   # the benchmark's required format

rows = [json.loads(l) for l in open("results/openai_agent_openai.jsonl")]
uncited = [r["answer"] for r in rows if not CITE.search(r["answer"])]
md_link = sum(bool(re.search(r'\]\(https?://', a)) for a in uncited)
no_url = sum(not re.search(r'https?://', a) for a in uncited)
print(len(uncited), md_link, no_url)   # 66 51 8

Re-scoring with every format counted

I wrote the new rule into the decision log before re-running anything: count [url], Markdown links [text](url) and the [[title](url)] bracket format as citations; don't count bare URLs in running text. Then I re-scored all ten runs, with both the original entailment model and an independent gpt-4o judge (chosen because it agreed best with my own blind judgements).

Three conclusions from that run, and they're less exciting than the ones above:

1. One run couldn't settle whether citations were really lost. The remaining gap was significant under one scorer and not the other, and it disappeared at temperature 0. At the time I wrote that a real loss wasn't established. (The next section is what settled it.)
2. The API switch and the published prompt fixed the format, not the citations. Counted in any format, neither changed whether sources got cited (measured on this run only). What they change is how often the model follows the requested format: 41 wrong-format answers drop to 9 (Chat Completions) or 20 (the published setup).
3. The robust result is the format drift itself, counted directly from the answers with no scorer involved: 51 of 127 on the SDK default, 0 on single-shot RAG.

I'd like to say I caught this before telling anyone. I didn't: I'd already sent the "loses half" version to people whose opinion I cared about, and had to follow up the same day with a correction. That was uncomfortable, and it was still the right call. (As the next section shows, the correction itself leaned too far the other way.)

One run couldn't settle it, so I ran everything three times

Everything above is one run per setup. The confidence intervals cover which questions happened to be in the set, but not how much a fresh set of generations moves the numbers. With a gap that was significant under one scorer and not the other, that run-to-run variation was exactly what I didn't know.

So, same habit as before, I wrote the rule down first: a difference counts as robust only if it has the same sign in all three runs and the pooled paired 95% interval excludes zero. Then I
ran every gpt-4o-mini setup twice more with fresh generations.

The format drift first, counted straight from the answers:

The drift is stable, and Chat Completions and the published setup cut it in every run. Temperature 0 on its own doesn't.

Then the question one run couldn't answer, with every format counted:

Per run, the SDK's gap was −0.163, −0.143 and −0.055 under the original scorer (−0.102, −0.087 and −0.045 under the judge). Same sign every time, under both scorers. So the SDK's default setup
does lose some citations for real. It's just much smaller than it first looked: counting only the requested format, the pooled gap is −0.302 [−0.394, −0.210]. About 60% of the apparent loss is format, and about 40% is a real, smaller loss of citations. At temperature 0, no loss is detectable.

Why one run couldn't settle it: the per-run gap ranged from −0.05 to −0.16. A single run can land anywhere in that range, and my first run happened to be the largest. The version I first sent overclaimed from it; my same-day correction, which called the gap not significant, underclaimed. The honest answer was in between, and it took three runs to see it.

A fix the retrieval server controls

Chat Completions and a good system prompt both reduce the drift, but both live in the customer's agent. A retrieval provider can't set them. So I tested a fix on the server side (pre-registered,
three runs per agent, on my self-hosted mirror server): the search tool tags each chunk with a short, stable source ID like [S3fa21c], the tool description and prompt ask for [ID] citations, and a small deterministic step maps each ID back to its URL after the answer is
written, accepting only IDs from that answer's own retrieved chunks.

On the SDK agent, wrong-format answers fell from 51 / 53 / 46 to 5 / 5 / 4, without the agent citing less. Any-format CSR rose by +0.096 (original scorer) and +0.125 (gpt-4o judge), and the
gap to single-shot RAG closed (−0.028 / +0.042, not significant). The details and the caveats (tool output and prompt changed together, no human audit of those citations yet) are in Part 6 (coming soon).

What I'd do differently

Read the raw outputs before building the theory, and certainly before sharing a number. I had the 66 answers from day one. A twenty-minute read would have pointed at format compliance before I
spent days on mechanism experiments, and before the overstated version left my laptop. The experiments were still worth it (they found the API effect on format and ruled out the loop and the tool description), but I'd have framed them correctly from the start.

Treat the parser as part of the metric. My scorer was validated against human judgement, three ways. All three checks shared the same citation parser, so none of them could catch this.

Check "same settings" line by line. The temperature confound sat in my code for two months.
It didn't change the conclusion, but I only know that because I tested it.

Run a close comparison more than once before calling it either way. One run made me overclaim first and underclaim second.

Takeaways for agent builders

1. Your framework's default model API is part of your system's behaviour. Two agents with the same model name followed the same citation instruction very differently.
2. Test format compliance explicitly. If anything downstream parses model output by format (citations, JSON, tags), count how often the format is followed, per framework and per API.
3. Put the rules where they work. A system prompt with explicit citation rules halved the format drift (41 → 20). The same intent in a tool description did nothing.
4. Measure format and presence separately. "No citation in the format I parse" and "no citation at all" are different failures with different fixes.
5. Write the experiment plan down before you run it. It's cheap, and it's the difference between "I found the cause" and "I kept trying things until one worked".

Limits: three runs per gpt-4o-mini setup (the tool-description run, E2, only once), one model (gpt-4o-mini), one corpus, and a self-hosted mirror of an MCP server rather than anyone's production system. The API switch and the published prompt were checked against any-format CSR on one run only. Whether the Responses API format drift holds for other models, or other agent frameworks, is the obvious next experiment.

The series: Citations at the MCP Boundary

1. Your MCP Server Returns the Source. Does the Answer Still Cite It?
2. One Agent Framework Seemed to Lose Half Its Citations. Most Were Just Reformatted. (you are here)
3. When Your Eval Grades Its Own Homework (coming soon)
4. I Audited My Own Benchmark Like a Hostile Reviewer. It Had Five Bugs. (coming soon)
5. Before You Measure an Agent, Fix Your Corpus: 20% of My Chunks Had No Code (coming soon)
6. Serving RAG Over MCP? A Practical Playbook for Keeping Your Citations (coming soon)

Previous: Part 1: Your MCP Server Returns the Source. Does the Answer Still Cite It?
Next: Part 3: When Your Eval Grades Its Own Homework (coming soon)

A new part is published every week.

Repo: https://github.com/AyanArshad02/csr-mcp-benchmark (results/sdk_ablation_report.txt,
results/anyformat_report.txt, results/format_compliance.txt and results/repeats_report.txt
have every number in this post).

I'm an AI engineer who builds production RAG and agent systems and the evals that prove they work, and I'm looking for a remote AI engineering role at a US or EU startup. If that's your team, I'd love to talk.