用 Python 提取 sitemap.xml 全部網址並追蹤競爭對手的新頁面

A site's sitemap is the cheapest full list of its pages you will ever get: the site publishes it for search engines, it usually carries a lastmod date per URL, and reading it takes seconds instead of crawling HTML for hours. That makes it the right starting point for SEO audits, migrations, content inventories, feeding a crawler or RAG pipeline, and keeping an eye on what competitors publish.

The catch is that "the sitemap" is rarely one file. Here's what you actually run into, and a small Python script that handles it.

What real sitemaps look like

• Discovery: the reliable place to look is robots.txt, which lists sitemaps on Sitemap: lines (often several). If it has none, try the usual paths: /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml (WordPress core), /sitemap.xml.gz.
• Sitemap indexes: big sites publish a <sitemapindex> that points to more sitemaps, and indexes can point to other indexes. Yoast, Rank Math, Shopify and most frameworks split by post type or by 50,000 URLs (the per-file limit in the protocol).
• Gzip: .xml.gz files are common on news and e-commerce sites. Don't trust the Content-Type header; check for the gzip magic bytes 1f 8b instead.
• Namespaces: every tag is namespaced ({http://www.sitemaps.org/schemas/sitemap/0.9}loc), and image, video, news and xhtml:link hreflang extensions add more. Strip the namespace and match on the local name.
• No sitemap at all: some well-known sites don't publish one. In my test python.org returned nothing from robots.txt or the common paths, so you need a crawler for those.
• Loops and duplicates: some indexes list themselves, and the same URL can appear in several sitemaps. Track what you've visited and dedupe.

The script

Only dependency: pip install requests. Standard-library xml.etree does the parsing.

import gzip, json, pathlib, re, sys, time
import xml.etree.ElementTree as ET
from urllib.parse import urljoin
import requests

UA = {"User-Agent": "sitemap-tutorial/1.0"}
COMMON = ["/sitemap.xml", "/sitemap_index.xml", "/wp-sitemap.xml", "/sitemap.xml.gz"]

def find_sitemaps(site):
    site = site if site.startswith("http") else "https://" + site
    found = []
    try:
        r = requests.get(urljoin(site, "/robots.txt"), headers=UA, timeout=20)
        if r.ok:
            found = [l.split(":", 1)[1].strip() for l in r.text.splitlines()
                     if l.lower().startswith("sitemap:")]
    except requests.RequestException:
        pass
    if found:
        return found
    for path in COMMON:  # fall back to the usual locations
        u = urljoin(site, path)
        try:
            r = requests.get(u, headers=UA, timeout=20)
            if r.ok and b"<" in r.content[:200]:
                return [u]
        except requests.RequestException:
            pass
    return []

def fetch_xml(url):
    r = requests.get(url, headers=UA, timeout=30)
    r.raise_for_status()
    body = r.content
    if body[:2] == b"\x1f\x8b":  # gzipped, whatever the Content-Type says
        body = gzip.decompress(body)
    return ET.fromstring(body)

def local(tag):  # strip the XML namespace: {http://...}loc -> loc
    return tag.rsplit("}", 1)[-1]

def walk(url, seen=None):
    seen = seen if seen is not None else set()
    if url in seen:  # some indexes list themselves
        return
    seen.add(url)
    root = fetch_xml(url)
    if local(root.tag) == "sitemapindex":
        for sm in root:
            loc = next((c.text.strip() for c in sm if local(c.tag) == "loc" and c.text), None)
            if loc:
                yield from walk(loc, seen)
    else:
        for u in root:
            row = {local(c.tag): (c.text or "").strip() for c in u if local(c.tag) in ("loc", "lastmod")}
            if row.get("loc"):
                yield row

def all_urls(site):
    urls = {}
    for sm in find_sitemaps(site):
        try:
            for row in walk(sm):
                urls[row["loc"]] = row.get("lastmod")
        except Exception as e:
            print(f"  skipped {sm}: {e}", file=sys.stderr)
    return urls

def new_since_last_run(site, urls, state=pathlib.Path("seen.json")):
    seen = json.loads(state.read_text()) if state.exists() else {}
    old = set(seen.get(site, []))
    seen[site] = sorted(urls)
    state.write_text(json.dumps(seen))
    return [u for u in urls if u not in old] if old else []

if __name__ == "__main__":
    for site in sys.argv[1:]:
        t = time.time()
        urls = all_urls(site)
        print(f"{site}: {len(urls)} URLs in {time.time() - t:.1f}s")

Run it with one or more sites:

python3 sitemap_urls.py crawlee.dev docs.apify.com allbirds.com stripe.com

What I got just now (October 20, 2026):

Gzip works the same way: walk("https://www.nytimes.com/sitemaps/new/news.xml.gz") returned 621 news URLs in 0.2 s, the newest with a lastmod only minutes old.

One thing to watch: robots.txt lives at the root, so asking for wordpress.org/news gave me 189,145 URLs for the whole of wordpress.org in 53 s. Filter on the path afterwards ([u for u in urls if "/news/" in u]) when you only care about one section.

Spotting new pages on a schedule

new_since_last_run() saves the URL set per site in seen.json and returns only the URLs it hasn't seen before. The first run records a baseline and returns nothing; after that, run it daily (cron, GitHub Actions) and you get a list of a competitor's new blog posts, product pages or docs:

urls = all_urls("competitor.com")
for u in new_since_last_run("competitor.com", urls):
    print("NEW:", u, urls[u])

To check it, I removed five URLs from the saved state and re-ran: exactly those five came back as new.

Ideas for extending it:

• Filter by lastmod ("changed in the last 7 days") to find refreshed content, not just new content.
• Read xhtml:link alternates to audit hreflang coverage across languages.
• Feed the URL list into a bulk PageSpeed or broken-link check.
• Send the new URLs to Slack or a Google Sheet.

If you'd rather not run it yourself

Disclosure: I'm with Datawright Labs, and we packaged this (plus text, RSS and Atom sitemaps, hreflang, image and Google News fields, URL and date filters, and the new-URL monitoring) as a hosted tool on Apify: Sitemap URL Extractor. It costs $0.20 per 1,000 URLs, which Apify's free monthly credit covers for most small jobs. You can run it on a schedule and get results by API, webhook or CSV. The script above is the whole idea, though, so use whichever fits.