Python 爬蟲如何遵守 robots.txt 和 Crawl-delay

A polite scraper checks robots.txt before each host, identifies itself, and slows down. This post shows how to do that with Python's standard library, and how the small open-source scraper in python-automation-starter handles it.

What robots.txt is, and is not

robots.txt is a plain-text file at the root of a site (https://example.com/robots.txt) listing which paths automated clients should avoid, and sometimes a Crawl-delay. It is a convention, not access control and not a legal permission. Terms of service, copyright and privacy law still apply. If a site says no, the right move is to skip it.

A sensible policy

1. Fetch robots.txt once per host, with your own User-Agent, and cache it.
2. On HTTP 200, parse the rules.
3. On a 4xx (for example 404), treat everything as allowed. That is the usual reading.
4. On a 5xx or a network failure, assume everything is disallowed, to be conservative.
5. Skip disallowed URLs and log why.
6. If the file has a Crawl-delay larger than your own delay, raise your delay to match.

Test the parser offline

You can see exactly how Python interprets a robots.txt without any network:

from urllib import robotparser

rp = robotparser.RobotFileParser()
rp.parse("""User-agent: *
Crawl-delay: 5
Disallow: /private/""".splitlines())

print(rp.can_fetch("MyBot", "https://x.test/private/a"))  # False
print(rp.can_fetch("MyBot", "https://x.test/ok"))         # True
print(rp.crawl_delay("MyBot"))                            # 5

Output when run: False True 5.

Set your identity and delay

Use a User-Agent that lets site owners contact you:

python static_list_scraper.py https://quotes.toscrape.com/ -c configs/quotes.json -o quotes.csv \
  --user-agent "MyResearchBot/1.0 (contact: me@example.org)" --delay 2

A 1-second delay is a minimum, not a goal. For anything bigger than a few pages, use 2 to 5 seconds, and scrape at quiet hours if you can.

What not to do

• Do not use an "ignore robots" switch to get around a block. Keep it for your own sites and test servers.
• Do not rotate User-Agents or IPs to hide your scraper.
• Do not hit a single host with parallel requests unless the owner agrees.
• Prefer an official API or data export when one exists.

Common errors

• Everything blocked on a site that looks fine: its robots.txt returned a 5xx error. Retry later.
• Rules differ by bot name: a User-agent: MyBot section applies only if your User-Agent string starts with that name. Otherwise the * section applies.
• Crawl-delay ignored in your own code: urllib.robotparser only reports it; you must sleep yourself.

The full code (scraper + retry/backoff + CSV/JSON export) is MIT on GitHub: https://github.com/raxsoft-sudo/python-automation-starter

If you'd rather not run it yourself, the same CSS-selector extraction is a hosted API with a free tier: https://rapidapi.com/%EA%B0%95%EC%A0%95%EC%8B%9D/api/scrape-to-json1

This post was written with AI assistance and the code was run and checked before publishing.