多數 AI 爬蟲不會執行 JavaScript

Every GEO article tells you to write helpful content and add schema. Fine advice. It also quietly assumes the machines can read your page, and for a large number of sites that assumption is wrong.

AI search runs in two steps. First the engine crawls and stores what it can read. Then, when somebody asks a question, it pulls the best passages and assembles an answer. The failure happens in step one, and it happens before any content tactic has a chance to matter.

What the crawlers actually do with JavaScript

Vercel ran a study in 2024, widely reported through 2026, logging how the major AI crawlers behave:

• GPTBot (OpenAI) fetched JavaScript files only sometimes and executed none of them.
• ClaudeBot (Anthropic) downloaded JavaScript in roughly a quarter of requests and executed none.
• PerplexityBot read plain HTML only.
• Googlebot is the exception. Gemini and AI Overviews go through Googlebot's renderer, so JavaScript does execute, on a delay.

So for three of the four, anything your site builds in the browser is invisible. Product specs pulled in by a script. FAQs that expand through a framework. Review counts fetched after load. None of it is there as far as the crawler is concerned.

How to check in thirty seconds

Open the page. Use view source, not inspect element. Inspect shows you the live DOM after JavaScript has run. View source shows the raw HTML response, which is roughly what a non-rendering crawler receives.

Search that source for a sentence you can see on screen. If it is in there, you are fine. If it only appears in the inspector, you have found your problem.

Or just fetch it:

curl -s https://yoursite.com/page | grep -i "a sentence from your page"

Nothing back means nothing for the crawler to cite.

The fix, in order

Server render the content. Static generation, server side rendering, or pre-rendering all solve it. Pick whichever suits the stack. The requirement is only that the real content is in the first HTML response.

Open the right crawlers in robots.txt. You cannot be cited by a system you blocked:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Two honest notes. PerplexityBot has been reported ignoring robots.txt, so a block there is not a guarantee of anything. And Google-Extended controls Gemini and training separately from normal Googlebot, so blocking it does not remove you from Google Search but does opt you out of some AI features. Make that call deliberately rather than by default.

Write in blocks. Retrieval systems do not read top to bottom, they pull pieces. Write so that any single section dropped onto a blank page with no title would still make sense. Lead with a 40 to 60 word answer, then explain underneath.

Add the evidence. The Princeton and IIT Delhi GEO study found that citing sources, adding quotes, and including statistics each raised visibility by roughly 30 to 40 percent. Keyword stuffing did nothing. Specific numbers with sources are the lever.

Ship entity signals. Organization schema with sameAs pointing to your verified profiles, consistent naming everywhere, a Wikidata entry if you qualify. Ahrefs found brand mentions across the web were the strongest correlate of AI visibility, ahead of backlinks.

Add Article and Product schema, and set honest last-updated dates. Ahrefs also found AI platforms preferred content roughly 26 percent fresher than normal search results favoured, and that most of ChatGPT's top cited pages had been updated within the previous 30 days.

Two things to not bother with

FAQ rich results. Google removed them from Search entirely in May 2026 and dropped the reporting and testing support that June. Keep the FAQPage markup, because any system reading JSON-LD still gets a clean description of your questions and answers. Just do not add it expecting the dropdown. It is gone for everyone.

llms.txt. A seven month server log study running from late 2025 into 2026 recorded zero requests to llms.txt from GPTBot, ClaudeBot, PerplexityBot, or Google-Extended. Google's John Mueller dismissed it in mid 2026. Add the file if you like a tidy site. Do not count it as a tactic that currently does anything.

The short version

Confirm your content is in the raw HTML. Server render whatever is not. Open the crawlers. Then worry about content.

Do those two things and you have cleared a bar most sites never clear.

Originally published as the Technical GEO Playbook on Techno Accelerator, where the full version includes the build order and validation steps.