跳到正文
DEV Community· AumCreate·· 1 天前AI 評分56

如何設定 AI 爬蟲可用的 sitemap 與逐頁 meta

A repeatable process for sitemaps and per-page meta that AI crawlers can actually use

AI 導讀

AumCreate 整理一套令 AI 爬蟲能發現、抓取、解析及引用網站內容的流程，涵蓋 sitemap、逐頁 meta、結構化資料和內部連結。流程要求 sitemap 只收錄狀態碼為 200、自我指向 canonical 且可索引的網址，並建議每個可索引頁面均可由首頁三次點擊內到達。文章亦涵蓋逐頁 meta、依範本加入結構化資料及定期核查，並提醒 llms.txt 仍是未獲採納的提議慣例。

正文

Most "AI SEO" advice is about prompts. The unglamorous truth is that an AI crawler has to do five boring things to your site before any of that matters: discover the URL, fetch it, parse the markup, chunk the content, and cite it. Each of those steps has a signal you control. This post is the process I run to make all five work.

It's ordered deliberately. Doing step 4 before step 1 is how you end up with a beautiful schema graph on a page that no crawler can reach.

Step 0 — Inventory before you touch anything

List every URL template on the site (post, page, product, category, tag archive, author archive, search, paginated). For each one write down: is it indexable, is it canonical, does it need to be discovered at all?

Sitemaps get bloated because nobody ever does this. A sitemap listing 4,000 tag archives is not a discovery aid, it's noise that dilutes the crawl budget you wanted to spend on the 200 pages that matter.

Step 1 — Fix status and canonical first

A sitemap entry is a promise: this URL is 200, canonical, and indexable. Break that promise and crawlers learn to trust the file less.

Rules:

  • Only 200 URLs go in the sitemap.
  • Only self-canonical URLs go in the sitemap. Never list a URL that canonicals elsewhere.
  • Never list a noindex URL. It's a contradiction — the page says "don't index me," the sitemap says "here's what to index."
  • Redirects don't go in. The destination does.

On WordPress, core already excludes most of this via wp-sitemap.xml, but check the tag and author archives — those are the usual leaks.

Step 2 — Build the sitemap properly

A minimal, valid file:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/guides/robots-txt/</loc>
    <lastmod>2026-10-14</lastmod>
  </url>
  <url>
    <loc>https://example.com/guides/sitemaps/</loc>
    <lastmod>2026-09-30</lastmod>
  </url>
</urlset>

Mechanics that matter:

  • Hard limits are 50,000 URLs and 50 MB uncompressed per file. Past either, split into a sitemap index.
  • lastmod must reflect a meaningful content change. Bumping it on every deploy trains crawlers to ignore the field entirely.
  • priority and changefreq are widely ignored. Skip them; they add noise without signal.
  • Keep the file UTF-8, well-formed, and free of BOM characters.

A sitemap index, for when you split by type:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-posts.xml</loc>
    <lastmod>2026-10-14</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-products.xml</loc>
    <lastmod>2026-10-12</lastmod>
  </sitemap>
</sitemapindex>

Step 3 — Advertise it in robots.txt and verify 

Sitemap: https://example.com/wp-sitemap.xml

That's the whole step — but verify it, because a broken sitemap URL in robots.txt is worse than none:

# 1. Is the sitemap well-formed XML?
curl -s https://example.com/wp-sitemap.xml | xmllint --noout - && echo "XML OK"

# 2. How many URLs, and do they look right?
curl -s https://example.com/wp-sitemap.xml \
  | grep -o '<loc>[^<]*</loc>' | sed 's/<[^>]*>//g' \
  | wc -l

# 3. Spot-check the first 50 for noindex (should return nothing)
curl -s https://example.com/wp-sitemap.xml \
  | grep -o '<loc>[^<]*</loc>' | sed 's/<[^>]*>//g' | head -50 \
  | while read -r u; do
      curl -sL "$u" | grep -qi 'name="robots" content="noindex"' && echo "NOINDEX: $u"
    done

That third check catches the most common sitemap defect in the wild: a plugin change turned a template to noindex and nobody noticed the sitemap still advertises it.

Step 4 — The per-page meta layer

Four tags per page, in this order of importance:

  1. <title> — unique, describes the actual page, no template boilerplate.
  2. <meta name="description"> — a real summary, ~150 characters. It gets used as snippet text and as chunk context.
  3. <link rel="canonical"> — self-referencing on every indexable page.
  4. <meta name="robots"> — only where it differs from the default. noindex,follow on search results and 404s.

WordPress outputs the canonical for you but not a description. A minimal, runnable addition:

add_action( 'wp_head', function () {
    if ( is_singular() ) {
        $id   = get_queried_object_id();
        $desc = wp_strip_all_tags( get_the_excerpt( $id ) );
        $desc = trim( preg_replace( '/\s+/', ' ', $desc ) );
        $desc = mb_substr( $desc, 0, 155 );

        if ( $desc ) {
            printf( "<meta name=\"description\" content=\"%s\">\n", esc_attr( $desc ) );
        }
    }

    if ( is_search() || is_404() ) {
        echo "<meta name=\"robots\" content=\"noindex,follow\">\n";
    }
}, 1 );

Swap get_the_excerpt() for whatever field holds your real summary if you have one — an excerpt built from the first 155 characters of body copy is a fallback, not a description. And if another plugin already emits a description, delete one of the two rather than shipping both.

Step 5 — Structured data, one shape per template

Schema is how you tell a parser what kind of thing the page is, instead of making it guess from prose. Match the type to the template — Article for posts, Product for products, BreadcrumbList site-wide, Organization once on the homepage.

add_action( 'wp_head', function () {
    if ( ! is_singular( 'post' ) ) {
        return;
    }

    $id = get_queried_object_id();

    $schema = [
        '@context'         => 'https://schema.org',
        '@type'            => 'Article',
        'headline'         => wp_strip_all_tags( get_the_title( $id ) ),
        'description'      => wp_strip_all_tags( get_the_excerpt( $id ) ),
        'datePublished'    => get_the_date( DATE_W3C, $id ),
        'dateModified'     => get_the_modified_date( DATE_W3C, $id ),
        'mainEntityOfPage' => [
            '@type' => 'WebPage',
            '@id'   => get_permalink( $id ),
        ],
    ];

    echo '<script type="application/ld+json">'
        . wp_json_encode( $schema, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE )
        . "</script>\n";
}, 20 );

Two rules: emit JSON-LD once per type (duplicate graphs from two plugins is a mess to debug), and only mark up what's visibly on the page. Marking up an FAQ that isn't rendered is the fastest route to losing rich results.

Step 6 — Internal links, because crawlers don't browse

Discovery isn't only the sitemap. It's also the link graph. A page with no inbound internal links is an orphan, and orphan pages are effectively undiscoverable no matter how clean the XML is.

  • Every indexable page reachable within three clicks of the homepage.
  • Descriptive anchor text — robots.txt patterns for AI crawlers, not click here.
  • Link from high-traffic posts down to the pages you actually want cited.
  • Check pagination: series and archives that only expose page 1 hide everything after it.

Step 7 — llms.txt, cheaply

llms.txt is a proposed convention, not an adopted standard — no guarantee any crawler reads it. Cost is near zero, so publish one, but don't build a strategy on it.

# Example Studio

> One-person WordPress studio. Templates, custom builds, deployment.

## Guides
- [robots.txt patterns](https://example.com/guides/robots-txt/): allowing search crawlers while blocking training crawlers
- [Sitemap setup](https://example.com/guides/sitemaps/): sitemap structure and per-page meta

## Optional
- [Changelog](https://example.com/changelog/)

Keep it to canonical URLs with one-line descriptions. A stale llms.txt full of 404s is worse than not having one.

Step 8 — The verification loop and its cadence

Per page, before publishing:

curl -sL https://example.com/guides/robots-txt/ \
  | grep -Eio '<link rel="canonical"[^>]*>|<meta name="robots"[^>]*>|<meta name="description"[^>]*>|<title>[^<]*</title>'

What you're looking for: one canonical, self-referencing; no stray noindex; a description that isn't just the title repeated.

Then on a schedule:

  • Weekly — new URLs appear in the sitemap, lastmod is honest.
  • Monthly — full sitemap sweep for 404s and noindex (the script in step 3, without the head -50).
  • On any template change — re-check the schema output and the canonical. Template edits are where noindex accidents happen.

Where the manual part stops

The process above is four files and a bash loop — which is fine for one site and miserable for five. This is exactly the boundary I hit, and why the sitemap, llms.txt, schema, meta and internal-link work on my own sites runs through AumViso instead of hand-maintained snippets.

But the tooling is the small part. The part that actually moves the needle is step 1: deciding which URLs deserve to be discovered, and then not contradicting yourself in the markup.

來源：DEV Community · dev.to

#教學/實踐#搜尋#部署/工程