Google 爬蟲推高 Because I liked 的 AI 費用

Because I liked asks what you loved and why, then finds films, shows, books and games that match the why. Part 1 was how it works; part 2 was measuring it.

This one is about money. Last week Anthropic charged my card $10, then $10 again, on days when almost nobody was using the site. My analytics showed a handful of searches. My model bill showed thousands.

Daily Haiku cost had been cents to about $2 a day. Then ~$7 on Oct 22 and ~$14 on Oct 23: two-thirds of the month's $34 in two days. Nobody had come. Something had.

The short version: the URLs that cost money were my own fault, and four of the five fixes only half worked.

🧾 How a search URL costs money

A search on the site is a few calls to Claude Haiku: one to understand the query (what medium, what era, what the person actually loved), one to write a line on why each top pick fits, one for picks in other media. Results are cached, so a search someone already ran is free. A new search is not.

Every title page also has "why chips": one-tap reasons like "shows like Breaking Bad: a decent man slowly turning bad". They were plain links:

<a href="/?q=shows+like+breaking+bad%3A+a+decent+man+slowly+turning+bad">

Now do the maths. About 13,850 title pages, several chips each, every chip a distinct search URL that nobody had run yet. I had built tens of thousands of uncached, paid, crawlable links, and listed every title page in my sitemap so crawlers would find them faster.

Crawlers did exactly what crawlers do. Each link they followed was a few Haiku calls.

🕷️ Fix 1: robots.txt is a request, not a wall

Google Search Console's crawl stats showed Googlebot fetching search URLs by the thousand. My search pages already carried noindex, and that doesn't help here: Google has to fetch the page to see the noindex, and the fetch is the part that costs money.

So search URLs went into robots.txt:

// src/app/robots.ts
disallow: [
  // Each new search is a few uncached Haiku calls, and the title pages'
  // why chips link to tens of thousands of them. They carry noindex
  // anyway, so nothing is lost in search.
  '/*?q=',
  // ...
],

Googlebot respects it. The bill didn't go to zero, because not everything that crawls is Googlebot.

🧱 Fix 2: firewalls stop the honest bots

What was left came in two kinds: a crawler on AWS IP addresses, and later something on rotating home IPs that ignored robots.txt entirely.

I added Vercel firewall rules (by user agent, and by AS number for the AWS traffic), then turned on Bot Protection, which challenges anything that doesn't behave like a browser. Each step cut the traffic. None got it to zero.

That's the limit of identifying bots by what they are. A scraper on rotating home IPs looks exactly like your users, because it's using the same kind of connection your users have.

(I also decided not to block bots in general. Search engines bring visitors, and link previews on X, Slack and WhatsApp come from bots too. Your /api/og route has to stay reachable or every shared link turns into a bare URL.)

🧯 Fix 3: a cost ceiling has to fail closed

Next, a ceiling: a site-wide daily cap on search-path model calls, counted in Redis.

// src/lib/daily-cap.ts
export async function takeSearchCall(): Promise<void> {
  if (!redis) return
  const key = `cap:search-llm:${new Date().toISOString().slice(0, 10)}`
  let used: number
  try {
    used = await Promise.race([
      redis.incr(key),
      new Promise<never>((_, reject) => setTimeout(() => reject(new Error('timeout')), 1_000)),
    ])
    if (used === 1) void redis.expire(key, 60 * 60 * 48)
  } catch (err) {
    throw new DailyCapReached('counter unavailable')
  }
  if (used > DAILY_CALLS) throw new DailyCapReached('cap reached')
}

Every model call on the search path takes from this allowance first. When it throws, the caller falls back: plain text search instead of understood search, posters without the "why it fits" line.

The interesting decision is in the catch. My per-IP rate limiter fails open: if Redis is down, let people search, because a broken rate limiter shouldn't take the site down. The cap fails closed: if the counter can't be read, skip the model call. A cost ceiling that disappears whenever Redis has a bad day isn't a ceiling.

And then Redis had a bad day. On launch day, my Upstash free tier hit its monthly command limit. Part of the reason was my own debugging: I'd scanned tens of thousands of cache keys while investigating all this. The cap did exactly what I designed it to do. It failed closed, and every search quietly fell back to plain text matching. Someone searched for something like The Sixth Sense and The Sixth Sense wasn't in the results.

Two lessons in one:

1. Fail closed is right for a cost ceiling, but it's only safe if you watch it. A fallback that's quiet enough to be a good user experience is also quiet enough to miss.
2. A cap can't tell a person from a bot. On a heavy scraper day, the bots spend the budget and the real users get the fallback. The cap limits the damage. It doesn't fix the cause.

🎟️ Fix 4: spend only on a person's action

The fix that actually worked came from changing the question. Not "who is this request from?", which I'd already lost, but "did a person just do something?"

A request may ask the model for something new only if:

• it's from a signed-in user (an account is a person), or
• it carries a valid search pass.

Otherwise the page is cache-only: anything already written is shown in full, and nothing new is asked of the model.

// src/lib/search-pass.ts
export async function modelCallsAllowed(signedIn: boolean): Promise<boolean> {
  if ((await headers()).get('next-router-prefetch')) return false
  if (signedIn) return true
  return validPass((await cookies()).get(PASS_COOKIE)?.value)
}

Router prefetches never spend money, pass or not. Next.js prefetches links in view, and a prefetch isn't a person.

The pass is a cookie: an expiry time and an HMAC signature over it, HttpOnly, valid for 30 minutes.

export function newPass(): string | null {
  const key = passKey()
  if (!key) return null
  const exp = String(Math.floor(Date.now() / 1000) + PASS_SECONDS)
  return `${exp}.${sign(key, exp)}`
}

export function validPass(value: string | undefined): boolean {
  const key = passKey()
  if (!key || !value) return false
  const [exp, sig] = value.split('.')
  if (!exp || !sig || !/^\d+$/.test(exp) || Number(exp) < Date.now() / 1000) return false
  const want = Buffer.from(sign(key, exp))
  const got = Buffer.from(sig)
  return want.length === got.length && timingSafeEqual(want, got)
}

No database, no session table. The signature means a client can't forge a pass or extend its expiry. timingSafeEqual means it can't guess one byte by byte either.

Only a Server Action sets the cookie, and the page calls it only on a person's action: submitting the search box, tapping a why chip, or the first tap or key press of a visit. The chips stopped being links:

<button
  type="button"
  onClick={() =>
    startTransition(async () => {
      await ensureSearchPass()
      router.push(href)
    })
  }
>

To a person, nothing changed: tap a chip, get a search. To anything that only follows links, the searches no longer exist.

A bot that loads a search URL directly now gets a "Ready when you are" page with a Show picks button, and it costs nothing. Searches that are already cached still render in full, so the pages search engines index lose nothing.

Is it a fortress? No. Someone could write a scraper for this site that calls the Server Action first. But that raises the bar from "fetch a URL" to "drive the page like a person", and the daily cap is still there as a backstop.

The result: it shipped on Oct 24. Oct 25 cost cents. I'm not counting the days after, because that's when Redis was out of commands and the cap was blocking everything, so they prove nothing. I'll update this post with a clean week.

📋 If you're putting an LLM behind a URL

1. Any GET that calls a model is a cost endpoint. If anything links to it, assume something will crawl it.
2. Make links to expensive pages buttons, or gate them behind a gesture. Crawlers follow hrefs. They don't click.
3. Separate "can read answers" from "can create new ones". Cache-only mode gave me both: full pages for search engines, no spend for bots.
4. Rate limiters fail open, cost ceilings fail closed. And alert on the one that fails closed, because it will be quiet.
5. Your diagnostics cost money too. Scanning cache keys used up a real slice of my Redis free tier.

The irony of this post: everything that went wrong here, I designed myself, including the clever chip links that made the product better for people and a feast for crawlers.

The site is Because I liked: describe what you loved and why, and get films, shows, books and games matched on the why. Try a search. You're a person, so this one's allowed to cost me money 🙂