跳到正文
DEV Community· PennyData·· 4 小時前AI 評分53

如何用 Python 下載網站上的所有圖片，並處理延遲載入、srcset 和 CSS 背景圖

How to download all images from a website with Python (lazy-loaded, srcset and CSS backgrounds)

AI 導讀

PennyData 分享一段 Python 腳本，用來擷取並下載網站圖片，涵蓋延遲載入、srcset、CSS 背景圖及社交圖片 meta 標籤。

正文

Right-click, "Save image as" does not scale past a handful of pictures. Here is a short Python script that downloads every image from a page, plus the traps that make naive scripts miss images or get you blocked.

The script

Install: pip install requests beautifulsoup4. Run: python grab_images.py https://books.toscrape.com/ images

#!/usr/bin/env python3
"""Download every image from a web page. pip install requests beautifulsoup4"""
import hashlib, os, re, sys, time
from urllib.parse import urljoin, urlparse
from urllib import robotparser
import requests
from bs4 import BeautifulSoup

UA = "image-grabber-tutorial/1.0 (+contact: you@example.com)"
CSS_URL = re.compile(r"url\(\s*['\"]?([^'\")]+)['\"]?\s*\)")
LAZY_ATTRS = ("src", "data-src", "data-lazy-src", "data-original", "data-lazy")

def best_from_srcset(srcset):
    """Pick the largest candidate from 'a.jpg 480w, b.jpg 1080w'."""
    best, best_w = None, -1
    for part in srcset.split(","):
        bits = part.strip().split()
        if not bits:
            continue
        w = 0
        if len(bits) > 1 and bits[1][:-1].replace(".", "").isdigit():
            w = float(bits[1][:-1]) * (1 if bits[1].endswith("w") else 1000)
        if w > best_w:
            best, best_w = bits[0], w
    return best

def find_images(html, page_url):
    soup = BeautifulSoup(html, "html.parser")
    found = []
    for img in soup.find_all("img"):
        for a in ("srcset", "data-srcset"):
            if img.get(a):
                found.append(best_from_srcset(img[a]))
        for a in LAZY_ATTRS:
            if img.get(a):
                found.append(img[a])
    for src in soup.select("picture source[srcset]"):
        found.append(best_from_srcset(src["srcset"]))
    for m in soup.select("meta[property='og:image'], meta[name='twitter:image']"):
        found.append(m.get("content"))
    blobs = [t.get_text() for t in soup.find_all("style")]
    blobs += [t["style"] for t in soup.find_all(style=True)]
    for blob in blobs:
        found += CSS_URL.findall(blob)
    urls = []
    for u in found:
        if u and not u.startswith("data:"):
            urls.append(urljoin(page_url, u.strip()))
    return list(dict.fromkeys(urls))  # dedupe URLs, keep order

def main(page_url, out_dir="images", min_bytes=5000, delay=1.0):
    s = requests.Session()
    s.headers["User-Agent"] = UA
    # fetch robots.txt ourselves: urllib's default UA gets a 403 on some sites,
    # and RobotFileParser then treats the whole site as disallowed
    rp = robotparser.RobotFileParser()
    rr = s.get(urljoin(page_url, "/robots.txt"), timeout=20)
    rp.parse(rr.text.splitlines() if rr.status_code == 200 else [])
    if not rp.can_fetch(UA, page_url):
        sys.exit("robots.txt disallows this page")
    html = s.get(page_url, timeout=20).text
    urls = find_images(html, page_url)
    print(f"{len(urls)} candidate image URLs")
    os.makedirs(out_dir, exist_ok=True)
    seen, saved = set(), 0
    for u in urls:
        if not rp.can_fetch(UA, u):
            continue
        time.sleep(delay)  # be polite
        try:
            r = s.get(u, timeout=30)
        except requests.RequestException as e:
            print("skip", u, e); continue
        ct = r.headers.get("content-type", "")
        if r.status_code != 200 or not ct.startswith("image/"):
            continue
        if len(r.content) < min_bytes:  # icons, tracking pixels
            continue
        digest = hashlib.sha1(r.content).hexdigest()
        if digest in seen:  # same file under another URL
            continue
        seen.add(digest)
        name = os.path.basename(urlparse(u).path) or digest
        name = re.sub(r"[^A-Za-z0-9._-]", "_", name)[:80]
        with open(os.path.join(out_dir, f"{digest[:8]}_{name}"), "wb") as f:
            f.write(r.content)
        saved += 1
        print(f"saved {len(r.content):>8} B  {u[:90]}")
    print(f"done: {saved} unique images in {out_dir}/")

if __name__ == "__main__":
    main(sys.argv[1], *(sys.argv[2:3] or ["images"]))

What it handles (and why)

  • Lazy loading. Many sites keep the real URL in data-src, data-lazy-src or data-original and put a 1px placeholder in src. The script checks all of them.
  • srcset and <picture>. One <img> can list a dozen sizes (a.jpg 480w, b.jpg 1080w). best_from_srcset keeps the widest candidate instead of the thumbnail.
  • CSS background images. url(...) inside inline style= attributes and <style> blocks. Not covered: external .css files (fetch those too if you need them).
  • Social images. og:image and twitter:image meta tags often hold the best-quality version.
  • Dedupe twice. URLs are deduped first, then files are deduped by SHA-1 of the content, because the same picture is often served under two URLs.
  • Size filter. min_bytes drops icons and tracking pixels. Raise it to 30000 for photos only.

Real output on two public sites

books.toscrape.com (a site made for scraping practice):

20 candidate image URLs
saved     9876 B  https://books.toscrape.com/media/cache/2c/da/2cdad67c44b002e7ead0cc35693c0e8b.jpg
saved     7095 B  https://books.toscrape.com/media/cache/26/0c/260c6ae16bce31c8f8c95daddd9f4a1c.jpg
...
done: 17 unique images in out_b/

20 URLs found, 17 unique files saved (the other 3 were dropped by the dedupe or the size filter).

en.wikipedia.org/wiki/Cat (heavy page with srcset everywhere):

112 candidate image URLs
saved   115918 B  https://en.wikipedia.org/static/images/icons/enwiki-25.svg
saved   34961 B   https://thumb.wikimedia.org/wikipedia/commons/thumb/2/25/Siam_lilacpoint.jpg/330px-Siam_li...
...
skip https://en.wikipedia.org/static/images/footer/wikimedia.svg ... Read timed out
done: 70 unique images in out_wp/

112 candidates, 70 unique files saved, one timeout skipped (the script logs it and carries on). It took about two minutes, because of the 1 second delay between requests, which is deliberate.

Traps

  1. robots.txt. The script checks it for the page and every image. A bug I hit while testing: Python's RobotFileParser.read() uses its own User-Agent, Wikipedia answered 403, and the parser then treated the whole site as disallowed, so the script refused to run. Fetching robots.txt yourself with requests and your own User-Agent, then parse(), fixes it.
  2. Rate limits. One request per second is polite. Hammering a site gets your IP blocked and is rude to the site owner. Set a User-Agent with a contact address.
  3. JavaScript-injected images. This script reads the HTML the server sends. Galleries that build themselves with JavaScript (infinite scroll, carousels fetched via API) need a real browser such as Playwright, or the JSON API the page calls (check the browser's Network tab).
  4. Hotlink protection. Some CDNs return 403 unless the Referer header matches the page. Add s.headers["Referer"] = page_url.
  5. Copyright. Being able to download an image does not mean you may reuse it. Check the site's terms and the image licence (Wikimedia Commons images carry per-file licences). Use downloads for personal archiving, analysis or with permission.

When one script is not enough

For dozens of sites, same-site crawling, size filters, or a ZIP of the files without maintaining the script, there is a hosted option. I run the Apify actor pennydata/website-image-extractor (plain HTTP, no browser, respects robots.txt). It reads img, srcset, picture, og:image and optionally CSS backgrounds, and returns one row per unique image.

20-second screen recording of a real run of website-image-extractor on a sample page

Screen recording of a real run (no audio).

Sample input:

{"mode":"images","startUrls":[{"url":"https://en.wikipedia.org/wiki/Cat"}],"maxImages":10}

Real output rows from run mvWmlc1aaw8J1Bdi8 (2026-10-25, 3 seconds, 10 rows, trimmed to three fields; run on the Wikipedia Cat page):

ok  Siam_lilacpoint.jpg                    image/jpeg  647001 bytes  source: og:image
ok  40px-Symbol_support_vote.svg.png       image/png     2206 bytes  source: img-srcset
ok  330px-Siam_lilacpoint.jpg              image/jpeg   34961 bytes  source: img-srcset

Same limits as the script: no JavaScript rendering and no login. Price at the time of writing: pay per event, USD 0.002 per image found on the free tier, failed or robots-blocked pages not charged. The actor page is the source of truth for current prices.

Questions or an image source the script misses? Comment below and I will test it.

I publish the Apify actor linked above and earn from its runs. Written with AI assistance (Claude).

來源：DEV Community · dev.to

#教學/實踐#編碼