跳到正文
DEV Community· William Jin·· 3 小時前AI 評分57

如何從網站 Sitemap 擷取所有 URL 並匯出 CSV

How to extract all URLs from a sitemap (and export them to CSV)

AI 導讀

William Jin 示範如何從網站 Sitemap 擷取所有 URL 並匯出 CSV，提供可處理巢狀索引、gzip 和重複網址的 Python、Node 腳本。文章説明如何透過 robots.txt 尋找 Sitemap，並處理 XML 命名空間、CDATA、特殊字元及大型索引等情況；另介紹作者製作的 Apify 託管工具及其 MCP 用法。

正文

You need a list of every page on a website. Maybe you're planning an SEO audit or a site migration. Maybe you want to check for broken links, or you're feeding pages into a crawler or an LLM pipeline. The fastest route is usually the XML sitemap: the site already publishes a list of its URLs for search engines.

The catch is that real sitemaps are rarely one tidy file. They're split into indexes, gzipped, capped at 50,000 URLs each, and sometimes hidden somewhere unexpected. This post covers how to find a sitemap and pull the URLs out by hand. Then come short Python and Node scripts that handle nested indexes and gzip, followed by the pitfalls that tend to bite.

Step 1: Find the sitemap

Try these in order:

  1. robots.txt. Open https://example.com/robots.txt and look for lines starting with Sitemap:. This is the most reliable source, and many sites list several sitemaps here.
  2. The usual paths. /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml (WordPress 5.5+), /sitemap.xml.gz.
  3. The CMS defaults. Shopify uses /sitemap.xml with child sitemaps like sitemap_products_1.xml. Yoast (WordPress) uses /sitemap_index.xml.
  4. Search Console, if it's your own site, lists the sitemaps you've submitted.

A sitemap comes in one of two shapes:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/about</loc>
    <lastmod>2026-09-14</lastmod>
  </url>
</urlset>

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://example.com/sitemap-posts.xml</loc></sitemap>
  <sitemap><loc>https://example.com/sitemap-pages.xml.gz</loc></sitemap>
</sitemapindex>

If you get a sitemapindex, the URLs you want are one level deeper, or sometimes two.

Step 2: Manual methods (small sites)

For a single, small, uncompressed sitemap you don't need code at all.

Google Sheets. In a blank cell:

=IMPORTXML("https://example.com/sitemap.xml", "//*[local-name()='loc']")

The local-name() trick sidesteps the XML namespace, which otherwise makes a plain //loc return nothing. Then use File → Download → CSV. Sheets times out on big files and can't follow indexes or open .gz, so this only works for small sites.

Command line. A quick and dirty version:

curl -s https://example.com/sitemap.xml \
  | grep -oE '<loc>[^<]+</loc>' \
  | sed -E 's#</?loc>##g' > urls.csv

This is fine for a one-off. It breaks on CDATA-wrapped URLs, on entities like &amp;, and on indexes. For a gzipped file, add | gunzip after curl (or use curl --compressed if the server only gzips the transfer).

Browser. Open the sitemap, press Ctrl+F, and copy. That works for 30 URLs. It doesn't work for 30,000.

Step 3: A script that handles indexes and gzip

Here's a small Python version using only requests and the standard library. It follows nested indexes, decompresses .gz files, de-duplicates URLs, and writes a CSV.

import csv, gzip, sys
import xml.etree.ElementTree as ET
import requests

HEADERS = {"User-Agent": "sitemap-extractor/1.0 (+https://example.com/contact)"}

def fetch_xml(url):
    r = requests.get(url, headers=HEADERS, timeout=30)
    r.raise_for_status()
    body = r.content
    if body[:2] == b"\x1f\x8b":            # gzip magic bytes, regardless of extension
        body = gzip.decompress(body)
    return ET.fromstring(body)

def local(tag):                             # strip "{namespace}"
    return tag.rsplit("}", 1)[-1]

def extract(url, seen_maps=None, depth=0):
    seen_maps = seen_maps if seen_maps is not None else set()
    if url in seen_maps or depth > 5:       # guard against loops
        return
    seen_maps.add(url)
    root = fetch_xml(url)
    kind = local(root.tag)
    for entry in root:
        fields = {local(c.tag): (c.text or "").strip() for c in entry}
        loc = fields.get("loc")
        if not loc:
            continue
        if kind == "sitemapindex":
            yield from extract(loc, seen_maps, depth + 1)
        else:
            yield {"url": loc, "lastmod": fields.get("lastmod", ""), "sitemap": url}

if __name__ == "__main__":
    start = sys.argv[1]
    seen = set()
    with open("urls.csv", "w", newline="") as f:
        w = csv.DictWriter(f, fieldnames=["url", "lastmod", "sitemap"])
        w.writeheader()
        for row in extract(start):
            if row["url"] not in seen:
                seen.add(row["url"])
                w.writerow(row)
    print(f"{len(seen)} unique URLs written to urls.csv")

Run it with python sitemap_urls.py https://example.com/sitemap_index.xml.

As a sanity check, I pointed it at https://docs.apify.com/sitemap.xml, which is a sitemap index with six child sitemaps. It wrote 4,082 unique URLs in a few seconds (the count will drift as the docs change).

The same idea in Node 18+ (it uses the built-in fetch), with fast-xml-parser as the only dependency (npm i fast-xml-parser, and save the file as .mjs so top-level await works):

import { XMLParser } from 'fast-xml-parser';
import { gunzipSync } from 'node:zlib';
import { writeFileSync } from 'node:fs';

const parser = new XMLParser({ removeNSPrefix: true });
const asArray = (x) => (x ? (Array.isArray(x) ? x : [x]) : []);

async function extract(url, out = new Map(), seen = new Set()) {
  if (seen.has(url)) return out;
  seen.add(url);
  let buf = Buffer.from(await (await fetch(url)).arrayBuffer());
  if (buf[0] === 0x1f && buf[1] === 0x8b) buf = gunzipSync(buf);
  const doc = parser.parse(buf.toString('utf8'));
  for (const s of asArray(doc.sitemapindex?.sitemap)) await extract(String(s.loc).trim(), out, seen);
  for (const u of asArray(doc.urlset?.url)) out.set(String(u.loc).trim(), u.lastmod ?? '');
  return out;
}

const urls = await extract(process.argv[2]);
const csv = ['url,lastmod', ...[...urls].map(([u, m]) => `"${u.replace(/"/g, '""')}",${m}`)].join('\n');
writeFileSync('urls.csv', csv);
console.log(`${urls.size} URLs written to urls.csv`);

Both versions check the gzip magic bytes rather than trusting the .gz extension. That matters more than you'd expect: some servers send .xml.gz files already decompressed, and some send gzipped bytes under a plain .xml name.

Nested sitemap indexes, in practice

The sitemaps.org protocol says an index should only list urlsets, not other indexes. In the wild you'll still find indexes that point to indexes, especially on big e-commerce and news sites that shard by date or category. So:

  • Recurse, but cap the depth (5 is plenty).
  • Track the sitemaps you've visited. Circular references happen, usually when an index lists itself.
  • Expect volume. One file can hold up to 50,000 URLs and 50 MB uncompressed, and an index can list 50,000 sitemaps. Large sites reach millions of URLs, so stream rows to disk instead of building one giant list in memory if you're going big.
  • Be polite. Fetch child sitemaps a few at a time, not hundreds in parallel, and back off on 429 or 503 responses.

Pitfalls

The XML namespace. root.findall("url") returns nothing because the real tag is {http://www.sitemaps.org/schemas/sitemap/0.9}url. Strip the namespace as the snippets do, or pass a namespace map.

Whitespace and CDATA. <loc> values are often padded with newlines, or wrapped in <![CDATA[ ... ]]>. Strip them. A real XML parser handles CDATA for you, and regex doesn't.

Escaped characters. &amp; in a <loc> is a literal & in the URL. Parsers decode it, and grep doesn't.

HTML instead of XML. Some sites return a 200 with an HTML "not found" page, or a styled HTML view of the sitemap. Check that the response actually starts with <?xml or <urlset/<sitemapindex before you parse.

Bot protection. Cloudflare and similar services sometimes challenge plain HTTP clients even on /sitemap.xml. Send a descriptive User-Agent, and respect the site's robots.txt and terms. If you're blocked, don't try to force your way through.

Text and RSS sitemaps. The protocol also allows plain-text sitemaps (one URL per line) and RSS/Atom feeds. If the parser fails, look at the first bytes of the file.

Stale sitemaps. A sitemap is what the site says it has, not what's actually live. For an audit, follow up by checking status codes. And lastmod is notoriously unreliable on some CMSs, which stamp every URL with the generation time.

Duplicates and variants. The same page can appear in several sitemaps, or with and without a trailing slash, www, or tracking parameters. Decide on a normalization rule before you de-duplicate.

CSV quoting. URLs can contain commas. Use a real CSV writer (as the Python version does) or quote every field.

If you'd rather not maintain a script

The scripts above cover most sites. Once you're doing this regularly, or across many domains, the edge cases add up: discovery through robots.txt, retries and backoff, huge indexes, text sitemaps, and filtering. Disclosure: I packaged all of that as a hosted tool, Sitemap URL Extractor on Apify, and the source is on GitHub.

You give it a domain or a sitemap URL. It discovers the sitemaps, follows nested indexes and .gz files, and returns one row per URL with lastmod, changefreq, priority and the sitemap it came from. Optional extras: include/exclude patterns, a lastmod date window (2026-01-01 or 7 days), an HTTP status check for every URL (to find 404s and redirects), and an "only new URLs since the last run" mode for scheduled monitoring.

The input is plain JSON:

{
  "startUrls": [{ "url": "https://docs.apify.com" }],
  "includeUrlPatterns": ["*/academy/*"],
  "maxUrls": 1000
}

maxUrls defaults to 1,000 so an accidental run on a huge site stays small. Set it to 0 for no limit.

From the command line (no SDK)

Apify's run-sync-get-dataset-items endpoint runs the tool and streams the results back in one request. This returns CSV directly:

curl -s -X POST \
  "https://api.apify.com/v2/acts/swillz~sitemap-url-extractor/run-sync-get-dataset-items?format=csv&fields=url,lastmod,sitemapUrl&maxTotalChargeUsd=0.50" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://docs.apify.com"}],"includeUrlPatterns":["*/academy/*"],"maxUrls":3}'

Output (trimmed):

"url","lastmod","sitemapUrl"
"https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actor-description",,"https://docs.apify.com/sitemap_base.xml"
"https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actors-and-emojis",,"https://docs.apify.com/sitemap_base.xml"

Note the empty lastmod: that sitemap simply doesn't publish one, which is common. maxTotalChargeUsd is a hard cap on what the run can cost you.

From Python

With the official client (pip install apify-client, tested with v3):

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("swillz/sitemap-url-extractor").call(
    run_input={
        "startUrls": [{"url": "https://docs.apify.com"}],
        "includeUrlPatterns": ["*/academy/*"],
        "maxUrls": 5,
    },
    max_total_charge_usd=0.05,
)
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item["url"], item.get("lastmod"))

On older 1.x versions of the client, call() returns a dict, so use run["defaultDatasetId"] instead.

Using it from an AI agent (MCP)

If you work with Claude, Cursor, VS Code or another MCP-capable client, you don't need to write any of the code above. Apify runs a hosted MCP server that exposes Store tools to agents. To give an agent just this one tool, add the server with a tools filter:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=swillz/sitemap-url-extractor"
    }
  }
}

On first connect it opens a browser window to sign in to Apify (OAuth). You can also pass an Authorization: Bearer <APIFY_TOKEN> header instead. Then you can ask in plain language:

"List every URL under /blog/ in example.com's sitemap that changed in the last 30 days, and flag any that return 404."

The agent maps that to startUrls, includeUrlPatterns: ["*/blog/*"], lastmodFrom: "30 days" and checkUrlStatus: true. The input fields carry descriptions and examples written for exactly this, so agents tend to get the call right the first time.

A few details that help in agent workflows:

  • Bounded by default. The 1,000-URL default keeps an agent from accidentally pulling a million-URL site. For run limits, the MCP call-actor tool also accepts maxItems and maxTotalChargeUsd in its call options.
  • No silent empties. If a site has no sitemap, or your filters matched nothing, the run says why (for example "0 results: No sitemap found for https://example.com in robots.txt or at /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and similar paths. Pass the sitemap URL directly if you know it.") instead of returning an empty list that the agent has to guess about.
  • Results by ID. call-actor returns the run and dataset IDs; the agent reads rows with get-dataset-items, which supports limit, offset and field selection, so it can page through large results instead of stuffing everything into context.

If you're building your own agent with LangChain, LlamaIndex or plain function calling, the curl call above is a good tool definition: one POST in, rows out.

Wrap-up

  • Start at robots.txt, then try the common paths.
  • Handle both urlset and sitemapindex, and recurse with a depth limit.
  • Detect gzip by magic bytes, not by file extension.
  • Strip namespaces and whitespace, and use a real XML parser and a real CSV writer.
  • Treat the sitemap as a claim, not the truth, and verify what matters.
  • For recurring jobs or AI agents, a hosted tool with sane limits saves you from maintaining the edge cases.

Happy crawling, and be nice to the servers you fetch from.

來源：DEV Community · dev.to

#教學/實踐#編碼#MCP/工具調用#Agent