跳到正文
DEV Community· LaunchSignal·· 4 小時前AI 評分54

我檢查 11 個產品發佈平台的 llms.txt:4 個有回應,1 個返回挑戰頁

I checked 11 launch platforms for an llms.txt. Four answered, one sent a challenge page

AI 導讀

作者檢查 11 個產品發佈平台及目錄的 /llms.txt,4 個返回 200 和 text/plain;Product Hunt 的 curl 請求遇到 Cloudflare 挑戰頁。十天內有 3 個平台的結果改變;文章把 403 挑戰頁與 404 缺檔分開判讀,並指出智能體能否讀取網站亦受抓取工具、請求來源和日期影響。

正文

On October 19 I ran a small, boring test: fetch /llms.txt from the root of 11 launch platforms and directories and see who answers. Ten days later, on October 29, 2026, I ran it again before writing this up. The picture had already moved.

llms.txt is a proposal for a plain markdown file at a site's root that tells a language model what the site is and where the useful pages are. Nobody is required to publish one, and Google's own guidance says you don't need new AI text files to appear in its AI features. So a missing file isn't a failing. I just wanted to know what an agent finds if it goes looking, because more of my own work now starts with an agent fetching a URL.

How I checked

One curl per domain, following redirects, with an ordinary desktop Chrome user agent:

curl -sL -A "Mozilla/5.0 ... Chrome/130.0.0.0 Safari/537.36" \
  -o /dev/null -w '%{http_code} %{content_type} %{url_effective}' \
  https://<domain>/llms.txt

I also tried /llms-full.txt, the longer companion file some sites publish. I counted a site as answering only when it returned a 200 with text/plain. A 404 counts as "no file". A 403 challenge page counts as "blocked the fetch", not "missing", because those are different things.

What came back

Platform /llms.txt What the file says
DevHunt 200, text/plain, 6 KB What DevHunt is, free listing vs. paid launch options, key pages, URL patterns for tool, alternatives and compare pages, categories. No submission API listed.
TinyLaunch 200, text/plain, 42 KB An "Agent Guide": a full submit flow over its REST API, email OTP auth, maker profile, startup, launch scheduling, categories, limits, error codes.
BetaList 200, text/plain, 3 KB What gets featured, links to an Agent API for drafting a submission without an account, which a human founder then claims; submission criteria and help pages.
Uneed 200, text/plain, 792 KB What Uneed is, a read-only MCP server and REST API for product data, a link to an agent launch guide (email OTP), then long lists of products, reviews and alternatives.
Product Hunt 403 to curl (Cloudflare challenge) Not readable with plain curl today. A different fetcher did get it: site overview, URL patterns, leaderboards, categories, and an attribution line.
Hacker News 404 No file
Fazier 404 No file
Open Launch 404 No file
SaaSHub 404 No file
Peerlist 404 No file
Microlaunch 404 No file

On /llms-full.txt: Uneed serves one (about 9 MB of text). TinyLaunch's path redirected to a login page, which returns 200 but is HTML, so I don't count it. Everyone else returned a 404 or the same challenge page.

What changed in ten days

Three results flipped since October 19:

  • BetaList returned a 404 on the 19th. Today it serves a short, well-organized file.
  • Uneed blocked my fetch on the 19th with a challenge page. Today it answered with the largest file in the set.
  • Product Hunt answered on the 19th. Today plain curl got a Cloudflare "Just a moment..." page with cf-mitigated: challenge, on both the apex and www. The file is still there; the same URL fetched through a different tool came back fine, with links dated today. I don't know why my request was challenged this time, only what came back.

That last one is the practical lesson. Whether an agent "can read" a site isn't a property of the site alone. It depends on who is asking, from where, and on which day. A single check is a snapshot.

Three kinds of file

Reading the four files that answered, they're doing different jobs.

The map. DevHunt's file, and Product Hunt's when it's reachable, explain what the site is and how its URLs are shaped: /tool/{slug}, /compare/{a}-vs-{b}, /products/{slug}/alternatives, daily and weekly leaderboard paths. That's what an assistant needs to answer "what are alternatives to X" by going straight to the right page instead of guessing. Product Hunt's also asks for something back: list a product's name and tagline first, and attribute the data to Product Hunt.

The runbook. TinyLaunch's file opens with "You are launching a startup on TinyLaunch on behalf of a user, end-to-end, without a browser." It then lists the eight endpoints the flow needs, which ones need a bearer token, and how to handle the email code. It even tells the agent to use its host's structured-question tool rather than printing options as text. That's written for a coding agent mid-task, not for a crawler.

The handoff. BetaList's file points to an Agent API where an agent drafts the submission and a human founder claims it. Uneed's does both a read side, an MCP server and REST API for its product data, and a write side, a separate launch guide for agents. I checked that the linked pages exist (betalist.com/agent and uneed.best/launch.txt both returned 200). I did not run either flow.

What none of the files did was hide the human step. Every write path I saw still routes through a person: an emailed code, or a founder claiming the draft. I think that's the right shape.

What the 404s mean

Not much on their own. Hacker News is a discussion site with an official read API documented on GitHub. The others have perfectly readable HTML pages, and a search crawler doesn't need a root markdown file to index them. The 404 only matters in one narrow case: an agent that checks /llms.txt first will learn nothing about how submissions work there and will fall back to reading the HTML or asking you.

What I'm taking from it

  • If you run a site that takes submissions, the most useful thing an llms.txt can carry is the submission path, plainly written, including the step that needs a human.
  • Write it for the agent that will act on it, not just the crawler that indexes it. TinyLaunch's step list and error codes are a good model of that.
  • Re-check anything you cite. Three of eleven results changed in ten days, and one of them seems to depend on how it was fetched.
  • Keep the raw responses. I saved every body and header from today's run, which is how I could tell a missing file from a challenge page.

The file I keep for LaunchSignal at launchsignal.ai/llms.txt is held to the same test: it names the JSON API and MCP submit paths, the error codes, and the fact that a submission stays pending until a person reviews it.

來源:DEV Community · dev.to