O’Reilly 的 Expert Intelligence 如何讓組織更聰明，而不只是更快

This event summary was created by Claude with light editing by Tim O’Reilly, based on a transcript of the event.

On October 15, 2026, Tim O’Reilly hosted Andrew Odewahn, CTO of O’Reilly Media, for an hour-long episode of Live with Tim O’Reilly, the interview series streamed live on the O’Reilly learning platform. Andrew has been at O’Reilly for many years, starting as an author, then building tools for the company’s platform, then eventually as CTO and head of engineering. Over the past year, his team has been building what the company calls Expert Intelligence, a suite of tools that connects an AI agent to an organization’s own systems, gives it a skill file describing how to reason about what it finds there, and grounds its analysis in the practitioner frameworks of the O’Reilly library. Andrew described the work in a recent Radar piece, “Building Organizational Intelligence.”

Tim and Andrew’s interview was organized around a presentation in two parts, the first on the research agent and the second on the system Andrew’s team built for reviewing what it produces, with discussion in between and audience questions throughout.

Tim framed the conversation by asking a question he said Andrew’s work helps to answer: Most of the AI story is about efficiency, doing the same work faster. What does it look like for an organization to get smarter?

Building the research agent

Andrew’s research agent emerged from a personal need rather than a strategy. He wanted to know the status of a project, for example, how long a migration from RabbitMQ to Pub/Sub would take. This is the kind of question every engineer dreads but every engineering leader has to answer. He also had a broader concern. A single person being good at AI isn’t the same as an organization being good at it.

He built his agent in four steps. He mapped the organization’s information systems as a hierarchy, connected them via MCP, wrote a skill telling the agent how to use them, and finally added the Expert Intelligence layer.

The map of the sources of truth in the engineering organization’s workflows was what made the rest work. Andrew put them in order and explained to the model how they relate: Strategic goals and feature priorities live in Productboard. Those become epics, sprints, and stories in Jira. Stories become code in GitHub. A service catalog describes the roughly 300 services that make up the O’Reilly platform and who owns them, and observability tools show how those services perform against their SLOs. The skill also had to encode local quirks. Some teams mark an epic as finished with an end date, while others consider it done when every ticket is closed. Andrew also told the agent it was a research agent, not a coding agent, and that its job was to write reports.

Connecting everything through MCP was, in his words, “the most completely tedious step of the whole process.” Productboard had no MCP server at the time, so he wrote a wrapper. Later, when Tim asked how another company could repeat the work with a different set of tools, Andrew suggested starting from the vendor list. Export it from your procurement system, ask the model what products those vendors make and which would hold information relevant to the questions you care about, and have it put them in order. O’Reilly product managers are now turning this process into a skill for the forthcoming Expert Intelligence Skills Hub.

One audience member asked whether MCP was really the right tool for internal knowledge, compared with semantic search. As Andrew pointed out, sometimes anything is better than nothing. Ultimately, what you want is to give the model the ability to search across your systems.

From a hiring case to a toil audit

The center of the presentation was a before-and-after comparison of the particular use case that Andrew had previously written up in his “Building Organizational Intelligence” article. An engineering manager asked Andrew to hire another engineer, and Andrew had his reporting and analysis agent make the hiring case. The first report covered the five-person team that manages content and metadata, the 10 tier-A services it owns along with its B and C services, the status of the Pub/Sub migration (64% complete), and the full backlog. It concluded that the team had a capacity problem and should probably hire. Andrew called it useful and very detailed. . .“maybe overly detailed.”

He then connected the agent to Expert Intelligence, asked it to act like a consultant, and ran the analysis again. The new report came to a very different conclusion. It opened by saying that the team’s operational overhead problem was “structural, not a staffing deficiency.” When it consulted Expert Intelligence, the agent brought in the concept of toil from Google’s Site Reliability Workbook. Toil is the manual, repetitive, automatable operational work that scales linearly with a service. The reporting and analysis agent used this notion to separate the team’s backlog into toil and the work people actually wanted to be doing. It recommended a structured toil audit before the next planning cycle, an explicit toil-reduction target, and treating runbook ownership as a first-class engineering artifact. Andrew said the team followed the recommendations and closed a substantial part of a backlog that had been sitting on everyone as a cognitive burden.

The report went on to cite Jim Highsmith, a signer of the Agile Manifesto and author of Wild West to Agile, to argue that the team’s technical debt was compounding. Andrew explained how that differed from the usual management advice:

A lot of the time when you read these high-concept frameworks, they’re like eating your broccoli. I should really be doing that. But what made this really interesting is that it pulled out a specific project we had, the Django upgrade, and put it within the context of this broader framework. It wasn’t just, “here’s an abstract principle you ought to be applying.” It’s “here’s your backlog, and here’s a specific example.”

A second analysis looked at services that crossed team boundaries. Drawing on microservices experts and Kent Beck’s work on coupling (see Tidy First?), it diagnosed the video metadata service as a complicated-subsystem problem rather than a boundary problem. In this version, Andrew had the agent pull out quotations, attribute them, and link to the source text. The resulting analysis was remarkably effective. The company has since retired a couple of systems. “I don’t need LLMs adding code,” Andrew argued. “I want them reducing it.”

Andrew said the part that surprised him most was the reaction from colleagues. His earlier reports were data dumps. Readers couldn’t tell where any given claim came from, and as a result they were much less actionable. When the analysis cited experts that engineers knew from their own work, the documents became, as he put it, “much more approachable, much more human,” and people engaged with them far more deeply.

Tim’s response: Invoking the experts

Tim broke in to put the demonstration in context. He contrasted it with the view that models are now smart enough that you can simply tell them to do something and let them rip, an approach we hear a lot about from vibe coders and those arguing for “dark factories.” Tim argued instead for a return to expertise.

He cited a post published that day by Felix Rieseberg of Anthropic, “How to Be More Ambitious: A Tutorial for Humans with Models,” which describes four kinds of ambition when working with models: speed, altitude, depth, and range. Most discussion of AI’s economic value is about speed, getting the same work done in less time. In Tim’s reading, Andrew had gained altitude (thinking about the outcome rather than the tasks) and depth (approaching the problem the way a field-leading expert would) simply by invoking the experts.

Tim then offered a metaphor he has been developing for a while, of AI programming as a kind of magic spellcasting. In the Marvel comic books and movies, Doctor Strange opens his spells by invoking some source, with words like “by the power of the dread Dormammu” or “by the hoary hosts of Hoggoth.” Andrew, he said, was doing the same thing, saying in effect, “By the power of Kent Beck, I want you to think differently about this problem.” Even where Andrew didn’t name an expert, as with the SRE workbook, simply pointing the model at the expertise embodied in the O’Reilly publishing canon raised the bar for what the agent produced.

Tim described this as using human expertise as a focusing mechanism, and connected it to his argument that LLMs are better understood as a medium we create with than as independent intelligences. He also pushed back on the idea, often drawn from “The Bitter Lesson,” that models will simply get better and human expertise will stop mattering. Andrew, he said, had found a particular way that it matters.

Andrew took up the metaphor. His own first prompt, he said, had been “Be snobby. Go find these great resources and look at them.” He wants to hand the model canonical works in a field, like Designing Data-Intensive Applications, and tell it “these are the ways I want you to think about it,” rather than accept whatever the model comes up with first. He compared it to business school, where much of what you learn is a set of frameworks for organizing facts.

What models do without direction

Tim argued that models don’t do their best work unprompted. Watch their chain of thought when they search for information, he said, and you may see them settle on some random file from the internet rather than canonical sources. Even with a frontier model like Fable, his own experience has been one of false starts and errors, a model brute-forcing its way to a solution rather than necessarily knowing how to approach it.

Andrew had a story from a few days earlier. He asked his agent to review incentive plans for the engineering and product teams.

The data that came back was, “according to the company Robusto (which I’m sure is a great company), this is the framework, and you should really be doing what they’re doing.” I’m like, “What is Robusto, and why am I seeing this?” And I realized, oh, I’m not connected to the VPN. So I hooked it up, and suddenly it was, “According to Harvard Business Review, these are the different ways you should be designing these incentive programs.” This was a very powerful model. But it keyed off the first thing it found and said, “This is the way I’m going to think about the problem.”

Tim raised an economic dimension. He pointed to instructions in Anthropic’s Claude system prompt that limit how much the model may quote from any source, and read them as a sign that AI companies want to avoid paying for content. The result, he argued, is that models can’t draw on the best material available to them specifically enough and too often fall back on generalities. He called this an early form of enshittification in AI.

O’Reilly takes the opposite position. Andrew explained that the company applies the royalty model it already uses for O’Reilly Answers. Material returned through the Expert MCP server can be cited, and authors are paid royalties on it, so that expert content has a market and earns a return.

In response to an audience question, Tim and Andrew noted that Expert Intelligence is in prerelease with a group of founding partners while O’Reilly works out its economics. Interested organizations can request access through the product page.

Putting the discussion where the AI can see it

The second part of Andrew’s presentation dealt with what happens after the agent writes a report. Decisions need consensus, and with AI-generated analysis people need to check that the model hasn’t keyed off something unimportant, out-of-date, or misinterpreted. “The AI can identify all the trade-offs,” he said, “but people still need to make the decisions that matter.”

His team first circulated reports as Google Docs, but they ran into two problems. The most interesting part of a document under review is usually the comment thread, and because at this time, O’Reilly company policy doesn’t allow MCP connections to Google Docs, the agent couldn’t see it. And when Andrew asked the model to fold the discussion back into the document, pasting the result wiped out the comments.

Doug Hogan, an O’Reilly engineering director, built an alternative on top of Git and GitHub that Andrew called SuperAnswers. (Andrew joked that this was why he isn’t allowed to name things.) Documents live in a Git repository, discussion lives in GitHub Discussions, and every change is versioned. Andrew can ask his agent what the consensus is on a project, have it incorporate comments into a new draft without losing history, and later reconstruct when a decision was made and what people knew at the time. The same record lets him generate the derivative documents a decision requires, such as scope statements for engineering, product, and design teams, or a meeting agenda. A graph view clusters the topics people are discussing and flags decisions that are still open.

Tim summarized the whole system as a research assistant connected to the organization’s sources of truth, a layer of outside expertise to put its findings in context, and a shared, versioned discussion that becomes an evolving store of organizational knowledge, which makes the AI a more useful coworker. Andrew agreed and said he would like to open-source SuperAnswers, which he described as a thin layer on top of Git.

Andrew also pointed out that the fragmentation he was addressing isn’t limited to engineering. Sales teams have their information spread across systems like Salesforce, 6sense, and Gainsight, and every function has the same need to pull those threads together. Tim added that tool access is uneven across the industry and always has been. Claude produces better PowerPoint than Google Slides, Gemini the reverse, and organizations have to capture that kind of knowledge about their own toolsets.

Questions from the audience

An audience member named Thomas worried that workers now talk to AI rather than to each other, and that work disappears into “that AI hole.” Andrew said his preference for keeping everything in Git is partly an answer to that, since it gives both humans and AI a central place to find and reconstruct past work. But his larger interest is in humans talking to humans in a way that AI supports. He described a category of questions he never used to ask:

I knew, if I ask this question, somebody’s got to go off for two weeks and compile all this stuff from all these different systems. Now I can gather it in a few minutes. Maybe some of this is throwaway, but it’s not bad if it’s cheap.

In response to that “throwaway” comment, one audience member wrote in the chat: “This is not throwaway work, Andrew. It is a product lead that changes the value proposition in a positive direction. Shit hot, bro.” Tim couldn’t help but read that one aloud!

Asked by Tim what he got wrong, Andrew conceded that he didn’t think early enough about how people would review and respond to AI-generated documents. People don’t want an AI-generated data dump to wade through, and they extend AI no benefit of the doubt. If an AI-written report gets one small fact wrong about Kubernetes operators, a reader is likely to dismiss the whole document. AI tools have to be understood as marshaling and presenting evidence, not making decisions. “These need to support your analytical capacity, not replace it.”

Tim connected that answer to concerns about deskilling. He described a fine-tuning project he attempted with Fable to train a small model to write more like him. He wanted to understand each stage rather than letting the model run, but it never explained them, and the training run failed. In part this was because the model asked him to provide samples of his writing, but it didn’t give him enough grounding on what a really good training dataset would look like, so the samples weren’t targeted enough. They were too long, and when Fable chose the short fragments it actually needed, it didn’t do an ideal job.

When Tim tried the same project with O’Reilly Tutor, a tool still in development, it sent him to read an O’Reilly resource on what a good training set looks like, and he realized where he went wrong. Tim compared the kind of deskilling that can happen when people just turn the work over to AI to people who can no longer read a map, or who call a plumber for a simple fix they could do themselves in 30 seconds if they only knew a bit more about how things work. He argued that everyone has an obligation to keep learning alongside the tools they use.

An audience member named David, a laid-off engineer looking to reskill, made a similar point. We used to know things, then we searched, and now we skip the search and just ask for the answer. He asked how to build real depth. Andrew suggested picking a project and working through every aspect of it, asking AI to act as a tutor (he’s relearning calculus that way) and keeping a book open alongside, or taking advantage of instructor-led live events on fast-moving topics.

The conversation ended with the difference between an AI system that simply makes an organization faster and one that makes it better at deciding what to do. In Andrew’s account, getting better at deciding what work needs doing depends on three things the models don’t supply on their own: a map of where the organization’s knowledge lives, a body of expertise that people trust, and a place where the humans can argue with what the AI produced.

You can watch full episodes of Live with Tim O’Reilly, including this one, on the O’Reilly learning platform, YouTube, or wherever you get your podcasts.