標普500非科技企業4,335場業績電話會議分析指出，關鍵字篩選大多會誤判AI主張

Apollo's chief economist argued in late October that AI has delivered "zero financial payoff" to S&P 500 companies outside tech, based on flat margins and revenue per worker. I wanted the firm-level version of that question: when executives outside tech talk about AI on their earnings calls, how often do they put a number on what it delivered?

The sample was every earnings call held since January 2024 by S&P 500 companies outside the tech sector, 4,335 calls from speaker-by-speaker transcripts. The headline:

Counting mentions is easy. Counting "put a number on AI" is not, and the gap between what a filter finds and what a person reading the sentence finds was the most useful lesson of the project.

1. The obvious filter has about 35% precision

The first pass looked for sentences by company speakers (not analysts) that mention AI and contain a number and a benefit word in the same or the next sentence:

import re

AI = re.compile(
    r"(?<![A-Za-z])A\.?I\.?(?![A-Za-z])"  # "AI" or "A.I.", not "AIG" or "said"
    r"|(?i:\bartificial intelligence\b|\bagentic\b|\bgen ?AI\b|\bLLMs?\b|\bmachine learning\b)"
)
NUM = re.compile(r"\$\s?\d[\d,.]*|\b\d[\d,.]*\s?(%|(percent|bps|million|billion|hours?|days?|times|x)\b)"
                 r"|\b(half|halved|doubled|two-thirds)\b", re.I)
BENEFIT = re.compile(r"\b(productiv\w*|efficienc\w*|sav\w*|reduc\w*|cut|faster|automat\w*|headcount)\b", re.I)

def candidates(sentences):
    for i, s in enumerate(sentences):
        if AI.search(s):
            window = s + " " + (sentences[i + 1] if i + 1 < len(sentences) else "")
            if NUM.search(window) and BENEFIT.search(window):
                yield window

The AI part is case-sensitive and allows "A.I.", the way some transcripts write it; the letter lookarounds keep it from matching inside names such as AIG.

It produced 534 candidate passages on 343 calls. Reading them, 122 calls had a real figure for what AI delivered. The rest were things a regex can't tell apart:

• Usage, not payoff. The share of staff who finished the training, the number of prompts in a month, the share of trucks with AI tools. A number next to AI, but it measures adoption.
• Spending. A company saying how many billions it put into AI and other technology. The money goes the wrong way.
• AI as demand. Utilities, data center landlords and materials companies talk about AI as customers: "data center and AI share of U.S. electricity consumption will triple". Real numbers, not a payoff from using AI.
• Bundled programs. A sentence reporting a large cost-savings total, then saying AI will speed up efficiency from here. The number and the AI sit together but are not connected.
• Speech recognition. A medical device company's "significant AI reduction" turned out to be AHI, the apnea index, misheard.

2. Then it misses about a quarter of the real ones

A second pass dropped the benefit-word requirement and widened it to revenue words (conversion, approval rates, growth, uplift), and listed every AI sentence with a number on calls the first pass hadn't flagged, or had flagged for a different sentence. It found 36 more calls.

The misses were mostly revenue and customer outcomes, phrased in ways a cost-savings vocabulary doesn't catch: an AI shopping assistant whose users convert at "triple" the rate (no digit at all), approval rates up 50 basis points from AI decisioning, an AI product line adding 300 basis points of organic growth. A few were cost claims split across sentences further apart than the window.

The lesson I'd keep: for a claim-level count, treat the regex as a way to rank what to read, not as the classifier. I read every candidate and every second-pass sentence; the labels took longer than all the code.

3. Check who you filtered out as "not the company"

A day after publishing I found two more holes, both in the filters rather than the reading, and both in the first pass and the second.

The first was the speaker filter. Transcripts give each turn a role, and I kept CEO, CFO, chair, COO and the like. Executives whose titles don't fit the role list, such as a chief commercial officer or a division president, are stored as "Other", and I had grouped "Other" with the operator. C.H. Robinson's results were mostly presented by those executives:

"In the AI agent's first few months, it has been determining the freight class and code for about 2,000 orders a day, and it has reduced the processing time from 10 minutes or more per shipment to 10 seconds or less."

The second was the window. A figure often comes first and the AI credit second. A sentence with both in it works, but "total non-standard surcharges are generating a significant year-over-year benefit of over $180 million annualized. This is the result of a new AI image capture process" puts the number one sentence before the AI. Fixing both added 385 passages to the first pass and 459 to the second, all read by hand. They added 65 calls with a figure, 223 in all, and raised last quarter's share from 6.5% to 9.0%. Turns with no role at all I only counted when reading them showed a company speaker, because some of them are analysts.

4. Decide what "credit" means before you label

Many figures credit AI along with other things: "efficiencies from organizational realignment, process redesign through AI and automation, procurement and vendor rationalization ... more than $1.5 billion". I labeled those separately from figures credited to AI alone. Last quarter 27 of the 35 calls with a number credited AI alone; most of the shared-credit ones are company-wide cost programs. Without that split, the count mixes an AI agent that resolves a stated share of support issues with a reorganization that mentions AI once.

5. Check for the same transcript on two calls

Before computing shares by quarter I hashed every transcript and grouped by company. Across the whole transcript archive, 182 transcripts were stored on more than one call event of the same company, 67 of them in different quarters; one company's investor webinar sat on five consecutive quarters' calls. In the S&P 500 sample there were 18 such pairs. Keeping one call per company per quarter handled the same-quarter ones, and I dropped the three pairs that crossed quarters:

-- PostgreSQL: identical transcripts on more than one call of the same company
with h as (
  select call_id, company_id, md5(string_agg(text, ' ' order by turn_index)) as hash
  from transcript_turns
  group by call_id, company_id
)
select company_id, hash, count(*) as calls
from h
group by company_id, hash
having count(*) > 1;

What the clean count shows

• Executives outside tech talked about AI on 61.2% of last quarter's calls and put a number on its payoff on 9.0%, about one in seven of the calls that mentioned it.
• Eleven companies account for 68 of the 223 calls with a number since 2024.
• The most common figure is a savings target for a company-wide program; the concrete results are narrow, in customer service, claims and document processing, and code.
• Analysts asked about AI savings, productivity or returns on 6.7% of calls last quarter, against about 1% in 2024.

The full write-up, with the charts, the sector table and every quote linked to its call: AI Talk Has Nearly Doubled Outside Tech

If you've labeled claims in transcripts or other long documents, did you find a way to get the filter's precision up, or did you also end up reading everything?

Data: S&P 500 earnings call transcripts, processed by Equibles.