跳到正文
DEV Community· Hilman Ramadhan·· 4 小時前AI 評分49

如何建立網頁爬蟲：初學者指南

How to build a web scraper: A beginner's guide

AI 導讀

入門指南説明如何以 Python、JavaScript 和 Ruby 建立網頁爬蟲，透過 HTTP 請求取得 HTML，再用 CSS 選擇器、XPath 或解析工具擷取資料。文章以 Hacker News 示範擷取貼文標題、外部連結及可見連結標籤，亦介紹 SerpApi 的 Extract API 作為較簡單的選項。

正文

The simple definition of web scraping is the automated process of extracting data from websites. It also usually involves saving that data into a structured format, like a database, CSV or JSON file, or text document. That way, you can parse large collections of pages (or entire websites) in a much more efficient way than a person browsing the pages and manually copying the desired information.

A web scraper first needs to make requests to a web server, and then fetch the contents of the requested page, just like a person using a web browser normally would. Then, it needs to parse the page into whichever format is required. Both of those steps can be tricky at scale—many servers will throttle or block IP addresses making bulk requests, and parsers have to understand various page layouts and rendering methods.

In this guide, we'll go over how to build your own web scraper in Python, JavaScript, and Ruby, and how to use the easier option—the Extract API from SerpApi.

Building your own web scraper

Web scraping can be done using a variety of programming languages and tools. Whether you're comfortable with JavaScript, Ruby, Go, or another language, libraries and frameworks are available to help you scrape websites.

The first step in web scraping is to fetch the HTML content of the web page you want. This is similar to how your web browser works when you visit a website—it sends a request to the server, and the server responds with the HTML, CSS, JavaScript, and other resources needed to display the page.

In web scraping, you'll use a programming language or tool to send an HTTP request to the website's server. This can be done using various methods:

  • Programming Libraries : Languages like Python, JavaScript, Ruby, and others have libraries that make it easy to send HTTP requests.
  • Built-in Functions : Some languages have built-in functions for making web requests.

By requesting the HTML content, you're retrieving the raw data that makes up the web page, which you can then process and analyze.

Once you've obtained the HTML content, the next step is to parse it to extract the information you're interested in. Web pages are structured documents that use HTML tags to define elements like headings, paragraphs, links, images, and more.

Some common methods of navigating through HTML pages are CSS selectors or XPath. For example, if you wanted to find all links in a page with "product" in the destination URL, you could use a CSS selector like this:

a[href*="product"]

Libraries like Beautiful Soup for Python, jsdom or Cheerio for Node.js, Nokogiri or Nokolexbor for Ruby, and others can parse HTML and XML documents. That way, you can navigate the document elements using standard selectors. Another option is using an AI model to parse the HTML.

Once you've identified the correct elements, you can extract the text, attributes, or other relevant information. That gives you the data for analyzing trends, feeding into another application, building a dataset, or other tasks.

Practice scraping a real website

It's time to build a web scraper! We will use the Hacker News home page as the example site, with the goal of extracting the title, external link, and visible link label for each post.

The Hacker News home page in a web browser.

First, we need to figure out where the links are in the page's HTML. Try right-clicking one of the links, then clicking the "Inspect Element" or "Inspect" menu option.

🧭 Using Safari on Mac? You may need to enable development features first. From the top menu bar, select Safari > Settings, click the Advanced tab, then check the box for "Show features for web developers."

This will open the HTML page's structure, and jump to where that link element is located.

The Elements view in Safari, showing the page structure.

As of the time of writing, each submission is a <tr> table row element with a athing CSS class. In each of those table rows, there's a .titleline element containing everything we need: a link element (a) with the external link and post title, and a .sitestr tag with the visible link label.

Now it's time to scrape that information!

Web scraping using Python

If you want to build a web scraper in Python, you can start with the Requests and Beautiful Soup libraries. For more complex web pages beyond this example, you can try Selenium or Scrapy.

First, install Python on your computer if you haven't already, then create the folder you want to use for this project. Next, open the Terminal (Mac/Linux) or PowerShell (Windows) application, and change the current directory to that project folder:

cd ~/Desktop/scraper

📁 Not sure how to find a folder's path? Type cd and a space, then drag the folder into your Terminal or PowerShell window.

Next, create a Python virtual environment for the project, so all dependencies and other data stays within the folder:

python -m venv venv

💻 Python command not found? Try "python3" instead. The name of the executable depends on how it was installed.

Now, activate the Python virtual environment. On a Mac with the default bash/zsh shell, run this:

source venv/bin/activate

On a Windows PC in PowerShell, run this:

venv\Scripts\Activate.ps1

Next, install the Beautiful Soup and Requests libraries in the project:

pip install requests beautifulsoup4

Now it's time to start writing code! Create a new Python script called main.py in your project with the following contents, then try running it with the python main.py command:

import requests

URL = 'https://news.ycombinator.com/'
response = requests.get(URL)

html_content = response.text
print(html_content)

This imports the Requests library, sends a GET request to the URL, and stores the HTML content in a variable.

Now, let's parse the HTML to extract the data. Add the following import statement to the top of the script:

from bs4 import BeautifulSoup

Replace the print(html_content) statement with this line, which creates a Beautiful Soup object for the parsed HTML content:

soup = BeautifulSoup(html_content, 'html.parser')

Next, the script needs to get a list of all table rows with the athing class, and extract each row's link URL, inner text, and visible link. You can do this with a soup.find_all function and a for loop at the end of the script:

rows = soup.find_all('tr', class_='athing')

for row in rows:
    titleline = row.find('span', class_='titleline')
    link_tag = titleline.find('a', href=True)

    title = link_tag.text
    link = link_tag['href']

    visible_link_tag = row.find('span', class_='sitestr')
    # This avoids an error when the visible_link_tag is None
    visible_link = visible_link_tag.text if visible_link_tag else "No visible link"

    print("Title:", title)
    print("Link:", link)
    print("Visible Link:", visible_link)
    print('---')

Running the entire script should give you an output like this:

Title: StreetComplete on iOS is now in public beta
Link: https://github.com/streetcomplete/StreetComplete/issues/5421
Visible Link: github.com/streetcomplete
---
Title: RIP, vector database
Link: https://turbopuffer.com/blog/rip-vector-database
Visible Link: turbopuffer.com
---
Title: How to speed up the Rust compiler in September 2026
Link: https://nnethercote.github.io/2026/09/30/how-to-speed-up-the-rust-compiler-in-september-2026.html
Visible Link: nnethercote.github.io
---

The website has multiple pages. If we want to grab all the data, we must first find the pattern. In this case, this is what the patterns look like:

You can check each page by wrapping the entire script in another for loop that counts from 1 to 3, and then runs the GET request with that number as the page count in the URL. The full script should now look like this:

import requests
from bs4 import BeautifulSoup

URL = 'https://news.ycombinator.com/'
max_pages = 3 # I gave the limit to only 3 pages

for i in range(1, max_pages+1):
    response = requests.get(f'{URL}?p={i}')
    html_content = response.text
    soup = BeautifulSoup(html_content, 'html.parser')

    rows = soup.find_all('tr', class_='athing')

    for row in rows:
        titleline = row.find('span', class_='titleline')
        link_tag = titleline.find('a', href=True)

        title = link_tag.text
        link = link_tag['href']

        visible_link_tag = row.find('span', class_='sitestr')
        visible_link = visible_link_tag.text if visible_link_tag else "No visible link"

        print("Title:", title)
        print("Link:", link)
        print("Visible Link:", visible_link)
        print('---')

The output will now include results from the first three pages.

You probably want to save the data somewhere other than the terminal output. Python's built-in CSV library can be used to write a CSV spreadsheet file, which can then be opened by Google Sheets, Microsoft Excel, LibreOffice Calc, and many other applications.

Here's how you could write a CSV file with the title, link, and visible link as the columns, and each post as a row:

import requests
from bs4 import BeautifulSoup
import csv

URL = 'https://news.ycombinator.com/'
max_pages = 3

# Open a CSV file in write mode
with open('data.csv', mode='w', newline='') as file:
    writer = csv.writer(file)

    # Write the header row
    writer.writerow(['Title', 'Link', 'Visible Link'])

    for i in range(1, max_pages+1):
        response = requests.get(f'{URL}?p={i}')
        html_content = response.text
        soup = BeautifulSoup(html_content, 'html.parser')

        rows = soup.find_all('tr', class_='athing')

        for row in rows:
            titleline = row.find('span', class_='titleline')
            link_tag = titleline.find('a', href=True)

            title = link_tag.text
            link = link_tag['href']

            visible_link_tag = row.find('span', class_='sitestr')
            visible_link = visible_link_tag.text if visible_link_tag else "No visible link"

            # Write the data row to the CSV file
            writer.writerow([title, link, visible_link])

Now, running the script will create a data.csv file in your project folder with all the data. The file looks like this when opened in a spreadsheet application:

A spreadsheet open in Apple Numbers, containing rows of titles, links, and visible links.

You now have a basic web scraper running in Python! The below resources can help you take on more complex scraping tasks:

You can also skip to the Extract API for a simpler method of web scraping.

Web scraping using JavaScript

JavaScript is also an option for web scraping projects, though not in the same way you might use the language in front-end development. Web browsers will block access to most external domains with CORS restrictions, which is required for scraping, so you need to use Node.js or another non-browser runtime. Those runtimes typically don't have a built-in Document Object Model (DOM), so common functions for parsing HTML like querySelector or getElementById() are unavailable unless you add additional libraries.

For this web scraping project, we'll use the Node.js runtime with the jsdom library. Node.js has a built-in Fetch API for retrieving web content, so additional libraries for that purpose (like Axios) are no longer required.

First, you'll need Node.js and the NPM package manager installed on your computer. Then create a folder somewhere for your project, open a Terminal window (Mac/Linux) or PowerShell prompt (Windows), and navigate to that folder. For example, this will change the current directory to a "scraper" folder on your Desktop:

cd ~/Desktop/scraper

📁 Not sure how to find a folder's path? Type cd and a space, then drag the folder into your Terminal or PowerShell window.

Next, we need to install the jsdom library in the project, with this command:

npm install jsdom

Next, create a new JavaScript file called main.js in the project folder, and start with this code that fetches the HTML for the home page and displays it in the console:

const response = await fetch("https://news.ycombinator.com");
const text = await response.text();
console.log(text);

You can run it with the node main.js command. Next, we can parse the HTML by importing the jsdom library, passing the text to a JSDOM constructor, and making a for loop that goes through every .athing table row:

import jsdom from "jsdom";

const response = await fetch("https://news.ycombinator.com");
const text = await response.text();
const dom = new jsdom.JSDOM(text);

// Get every table row representing a post
for (let el of dom.window.document.querySelectorAll("tr.athing")) {
    // Get the title, link, and visible link
    const title = el.querySelector(".titleline a").textContent;
    const link = el.querySelector(".titleline a").href;
    const visibleLink = (el?.querySelector(".sitestr")?.textContent || "No visible link");
    // Display the data
    console.log(`Title: ${title}\nLink: ${link}\nVisible Link: ${visibleLink}\n---`);
}

The website has multiple pages. If we want to grab all the data, we must first find the pattern. In this case, this is what the patterns look like:

This is a modified version of the script that fetches the first three pages, and then displays all links from all pages:

import jsdom from "jsdom";

const baseUrl = "https://news.ycombinator.com"

for (let i = 1; i <= 3; i++) {
    const response = await fetch(`${baseUrl}/?p=${i}`);
    const text = await response.text();
    const dom = new jsdom.JSDOM(text);
    // Get every table row representing a post
    for (let el of dom.window.document.querySelectorAll("tr.athing")) {
        // Get the title, link, and visible link
        const title = el.querySelector(".titleline a").textContent;
        const link = el.querySelector(".titleline a").href;
        const visibleLink = (el?.querySelector(".sitestr")?.textContent || "No visible link");
        // Display the data
        console.log(`Title: ${title}\nLink: ${link}\nVisible Link: ${visibleLink}\n---`);
    }
}

You might want to save the scraped data outside of the terminal output. We can write the results to a CSV file by importing Node's fs library and constructing the CSV format:

import jsdom from "jsdom";
import fs from "node:fs/promises";

const baseUrl = "https://news.ycombinator.com"
let csvText = `"Title","Link","Visible Link"`

for (let i = 1; i <= 3; i++) {
    const response = await fetch(`${baseUrl}/?p=${i}`);
    const text = await response.text();
    const dom = new jsdom.JSDOM(text);
    // Get every table row representing a post
    for (let el of dom.window.document.querySelectorAll("tr.athing")) {
        // Get the title, link, and visible link
        const title = el.querySelector(".titleline a").textContent;
        const link = el.querySelector(".titleline a").href;
        const visibleLink = (el?.querySelector(".sitestr")?.textContent || "No visible link");
        // Add the data as a new row
        csvText += `\n"${title}","${link}","${visibleLink}"`
    }
}

await fs.writeFile("data.csv", csvText);

Now, running the script will create a data.csv file in your project folder with all the data. The file looks like this when opened in a spreadsheet application:

A spreadsheet open in Apple Numbers, containing rows of titles, links, and visible links.

You now have a basic web scraper running in Node.js with JavaScript! The below resources can help you take on more complex scraping tasks:

You can also skip to the Extract API for a simpler method of web scraping.

Web scraping using Ruby

Ruby is another language option for building a web scraping tool. You can use libraries like Net::HTTP or HTTParty for making GET requests to web pages, Nokolexbor or Nokogiri for parsing HTML and XML elements, and Selenium for scraping with an automated web browser.

First, you need Ruby installed on your computer. You can download it from the official website or use a version manager like rbenv or RVM. Then create a folder somewhere for your project, open a Terminal window (Mac/Linux) or PowerShell prompt (Windows), and navigate to that folder. For example, this will change the current directory to a "scraper" folder on your Desktop:

cd ~/Desktop/scraper

📁 Not sure how to find a folder's path? Type cd and a space, then drag the folder into your Terminal or PowerShell window.

Next, create a plain text file called Gemfile in your project folder to keep track of dependencies, with source 'https://rubygems.org' as the only line. You can do that in a few different ways:

  1. Use a plain text editor (make sure not to add a .txt extension to the file)
  2. For Windows PowerShell, run this command in the project folder: Add-Content Gemfile "source 'https://rubygems.org'"
  3. For Mac/Linux, run this command in the project folder: "source 'https://rubygems.org'" >> Gemfile

We will use HTTParty for fetching web pages, Nokolexbor for parsing HTML, and csv for writing the data to CSV files. The CSV gem is usually bundled with Ruby, but it should still be defined as a dependency. Make sure they are installed with this command:

bundle add httparty nokolexbor csv

Next, create a Ruby file called main.rb in your project folder. The first version fetches the home page for Hacker News using the httparty gem:

require 'httparty'

url = 'https://news.ycombinator.com/'
response = HTTParty.get(url)
html_content = response.body
puts html_content

You can run it with the ruby main.rb command. Next, we need to add a require line for nokolexbor and then use it to parse each .athing table row as a post:

require 'httparty'
require 'nokolexbor'

url = 'https://news.ycombinator.com/'
response = HTTParty.get(url)
html_content = response.body
parsed_page = Nokolexbor::HTML(html_content)

parsed_page.css('tr.athing').each do |el|
    # Get the title, link, and visible link
    title = el.at_css('.titleline a').text
    link = el.at_css('.titleline a')['href']
    visible_link = el.at_css('.sitestr') ? el.at_css('.sitestr').text : 'No visible link'
    # Display the data
    puts "Title: #{title}\nLink: #{link}\nVisible Link: #{visible_link}\n---"
end

The website has multiple pages. If we want to grab all the data, we must first find the pattern. In this case, this is what the patterns look like:

Modifying the script to parse the first three pages isn't too difficult. We just need a loop that adds the current iteration to the page URL, like this:

require 'httparty'
require 'nokolexbor'

url = 'https://news.ycombinator.com/'
max_pages = 3

(1..max_pages).each do |i|
    response = HTTParty.get("#{url}?p=#{i}")
    html_content = response.body
    parsed_page = Nokolexbor::HTML(html_content)

    parsed_page.css('tr.athing').each do |el|
        # Get the title, link, and visible link
        title = el.at_css('.titleline a').text
        link = el.at_css('.titleline a')['href']
        visible_link = el.at_css('.sitestr') ? el.at_css('.sitestr').text : 'No visible link'
        # Display the data
        puts "Title: #{title}\nLink: #{link}\nVisible Link: #{visible_link}\n---"
    end
end

You might want to save the scraped data for future use. We can write the results to a CSV file by adding a require line for csv, creating a header row and a row for each post, and then writing it to a file:

require 'csv'

URL = 'https://news.ycombinator.com/'
max_pages = 3

# Open a CSV file in write mode
CSV.open('data.csv', 'w') do |csv|
  # Write the header row
  csv << ['Title', 'Link', 'Visible Link']

  (1..max_pages).each do |i|
    response = HTTParty.get("#{URL}?p=#{i}")
    html_content = response.body
    parsed_page = Nokogiri::HTML(html_content)

    rows = parsed_page.css('tr.athing')

    rows.each do |row|
      titleline = row.at_css('span.titleline a')
      title = titleline.text
      link = titleline['href']

      visible_link_tag = row.at_css('span.sitestr')
      visible_link = visible_link_tag ? visible_link_tag.text : 'No visible link'

      # Write the data row to the CSV file
      csv << [title, link, visible_link]
    end
  end
end

puts 'Data successfully written to data.csv'

Now, running the script will create a data.csv file in your project folder with all the data. The file looks like this when opened in a spreadsheet application:

A spreadsheet open in Apple Numbers, containing rows of titles, links, and visible links.

You now have a Ruby-powered web scraper! Check out these resources for help with more complex scraping tasks:

  • CSS selectors and combinators: Learn about using CSS selectors to find elements, which are used by Nokolexbor, Nokogiri, and other HTML parsing methods.
  • Selenium documentation: For sites that load dynamically instead of providing static HTML, you need to use Selenium or another browser automation tool.
  • Web scraping with Ruby: A more complete guide on web scraping methods while using Ruby.

You can also skip to the Extract API for a simpler method of web scraping.

Challenges with web scraping

Basic web scraping, with pages already rendered in HTML with consistent element names, isn't too complicated in many programming languages and environments. However, you will inevitably run into problems as you try parsing different websites, or try requesting many pages.

These are some of the common challenges with web scraping at scale:

  • Identifying data sources : Determining where and how data is stored can be complex. Websites may store data across different pages or make requests over APIs, making it difficult to pinpoint the exact source.
  • Handling CAPTCHA tests : Websites often use a CAPTCHA, short for "Completely Automated Public Turing test to tell Computers and Humans Apart," to block automated access. Bypassing these tests without violating terms of service requires sophisticated methods, such as using CAPTCHA-solving services, which adds cost and complexity.
  • Managing IP blocks and rate limits : Frequent requests from the same IP can lead to blocks or bans. Using proxies to rotate IPs and implementing smart throttling mechanisms can help mitigate this but require careful management and additional infrastructure.
  • Legal and ethical considerations : Scraping must be done in compliance with laws and website terms of service. Ignoring these can lead to legal actions and ethical issues.
  • Data structure changes : Websites often update their layout and structure, which can break scrapers. Regularly maintaining and updating scraping scripts is crucial to handle these changes.
  • Dynamically-rendered pages: Many websites will render the page with JavaScript, instead of sending a complete HTML structure over the initial request. This requires fetching the page with an automated web browser, like Selenium or Puppeteer, instead of a simple GET request.

The easy web scraping method

If you don't want to build and maintain your own web scraper, you can use our Extract API. It can parse any public web page and return its contents as structured JSON or Markdown, including automatic detection of headers, meta tags, inline links, and the complete page text. There's also a Map mode for finding all pages on a given website, and Crawl mode for finding and parsing all of a website's pages.

If you request a page on an engine or platform that has a dedicated SerpApi API, the results will be automatically routed to that API. For example, fetching a YouTube video URL with the Extract API will return all the data available through the YouTube Video API, like the video description and all recommended videos.

If the built-in parsing from the Extract API isn't enough, you can still use it to retrieve the fully-rendered HTML, and then build your own parser using that input.

Getting started with SerpApi

You need a free SerpApi account before you can use the Google Play Store APIs. You can upgrade to a paid account in the future if you need more search capacity, faster speeds, or other features.

First, create an account and verify your email and other details. After that, get the API key from your account dashboard.

The API key in the dashboard

You should store your API key in a safe location if you are sharing or publishing your code. If the key is leaked or stolen, you can make a new one from the SerpApi account dashboard.

Install the SerpApi library (optional)

You can use SerpApi's official libraries for Python, JavaScript, Ruby, Java, and other languages. They aren't required, but they provide a simple wrapper around the API requests.

The APIs can also be used with cURL, fetch() in Node.js, or anything else that can send GET requests and process a JSON response.

Extract API with a GET request

Here's how you could fetch the Hacker News home page with the Extract API, using a simple GET request:

https://serpapi.com/search.json?engine=extract&url=https://news.ycombinator.com&api_key=YOUR_KEY_GOES_HERE

The default response is in JSON format, with basic page information in the page_information object, a list of all included links in the inline_links object, and so on.

If you want the response in Markdown format, instead of JSON, just add output=md to the request:

https://serpapi.com/search.json?engine=extract&url=https://news.ycombinator.com&output=md&api_key=YOUR_KEY_GOES_HERE

If you need the page's rendered HTML, for building your own parser or some other use case, you can set output=html for the request:

https://serpapi.com/search.json?engine=extract&url=https://news.ycombinator.com&output=html&api_key=YOUR_KEY_GOES_HERE

Extract API with JavaScript

Here's how you can use the Extract API with the SerpApi JavaScript library to display all posts from the first Hacker News home page:

import { getJson } from 'serpapi';

let search = await getJson({
  engine: "extract",
  api_key: YOUR_KEY_GOES_HERE,
  url: "https://news.ycombinator.com/",
  mode: "scrape"
});

for (let link of (search?.inline_links || [])) {
  // Skip internal links, and the "API" link at the bottom of the page
  if (link.internal || link.link === "https://github.com/HackerNews/API") {
    continue;
  }
  // Display the title, link, and visible link text
  console.log(`Title: ${link.title}\nLink: ${link.link}\nVisible Link: ${link.displayed_link}\n----`)
}

The Extract API fetches the page contents and parses it, so all that is required is going through the list of extracted links (search.inline_links) and filtering the results to external URLs.

Here's a more advanced version that scans the first three pages, and saves the results to a CSV file:

import { getJson } from 'serpapi';
import fs from "node:fs/promises";

const baseUrl = "https://news.ycombinator.com"
let csvText = `"Title","Link","Visible Link"`

let settings = {
  engine: "extract",
  api_key: YOUR_KEY_GOES_HERE,
  mode: "scrape"
}

for (let i = 1; i <= 3; i++) {
  settings.url = `${baseUrl}/?p=${i}`;
  let search = await getJson(settings);
  for (let link of (search?.inline_links || [])) {
    // Skip internal links, and the "API" link at the bottom of the page
    if (link.internal || link.link === "https://github.com/HackerNews/API") {
      continue;
    }
    // Add the data as a new row
    csvText += `\n"${link.title}","${link.link}","${link.displayed_link}"`;
  }
}

await fs.writeFile("data.csv", csvText);

The crawl mode is helpful for analyzing many pages in a group. This example will crawl the UniFi store, up to 50 pages, to find every product page:

// import { getJson } from 'serpapi';

let search = await getJson({
  engine: "extract",
  api_key: YOUR_KEY_GOES_HERE,
  url: "https://store.ui.com/us/en",
  mode: "crawl",
  max_pages: 50
});

let productUrls = [];

// Iterate through all crawled pages
for (let page of search.pages) {
  // Iterate through all inline links
  for (let result of (page?.inline_links || [])) {
    // Filter out non-product links, and links already added to the list
    if (result.link.includes("/products/") && !productUrls.includes(result.link)) {
      productUrls.push(result.link);
    }
  }
}

// List all products
for (let product of productUrls) {
  console.log(product);
}

More information

You can learn more from the documentation pages for the Extract API, Map mode, and Crawl mode.

來源：DEV Community · dev.to

#教學/實踐#部署/工程