如何用 Python 建立本地化 SERP 監測流程
How to Build a Localized SERP Monitoring Pipeline with Python
教程示範如何用 Python 建立本地化 SERP 監測流程,涵蓋搜尋結果採集、解析、SQLite 儲存,以及按地點、語言、裝置和時間比較變化。文章建議保留原始回應,並把各項搜尋請求背景存為獨立資料,避免混合不同搜尋環境進行比較。SERP 數據亦可接入 AI Agent 或 RAG 工作流,並保留來源 URL、標題、摘要、擷取時間及搜尋查詢,以便審核。

Manually checking search results works when you have a few keywords. It stops working when you need to track dozens of keywords across multiple cities, languages, and search engines—especially when rankings, ads, and SERP features change daily.
This article walks through a practical architecture for a SERP monitoring pipeline: collecting search results, storing them in a comparable format, analyzing changes over time, and optionally feeding the results into an AI workflow.
This article was created with the help of AI.
Why SERP Monitoring Matters
A search engine results page, or SERP, is not just a list of blue links. It can include organic results, paid ads, featured snippets, local packs, image results, video results, and other modules.
For a business, the same keyword can mean very different things in different markets. A product page may rank well in one city but be invisible in another. A competitor may dominate a local search result. A new content format may start taking attention away from traditional organic listings.
A monitoring pipeline helps you turn those observations into repeatable data:
Track keyword rankings over time.
Compare results across cities, countries, and languages.
Detect new competitors or changes in the search landscape.
Monitor branded search results and potential reputation issues.
Study which SERP features appear for high-value queries.
Provide current search context to AI agents or RAG systems.
The goal is not to replace human judgment. It is to remove repetitive data collection so the team can focus on interpretation and action.
A Simple Pipeline Architecture
A useful SERP monitoring system does not need to be complicated. At a minimum, it needs four parts:
*Keyword configuration *— the queries, locations, languages, and schedules you want to monitor.
*Collection layer *— a scheduled job that requests search results and saves the raw response.
Parsing layer — code that converts the response into structured records.
Storage and reporting — a database, dashboard, or alerting system that makes trends visible.
A minimal architecture could look like this:
text
Keyword config
|
v
Scheduler
|
v
SERP request layer
|
v
Raw response storage
|
v
Parser
|
v
Structured database
|
v
Dashboard / alerts / AI workflow
The most important design decision is to always retain the raw response. Parsers change, search engines change their page structure, and you may later discover a data point you did not originally plan to extract.
Design the Data Model First
Before writing code, decide what you want to compare.
A common mistake is storing only the keyword and its ranking position. That makes it impossible to answer useful questions later, such as whether a result changed because of location, device, language, or time.
A practical table might include:
text
search_requests
- id
- keyword
- search_engine
- country
- city
- language
- device
- requested_at
- raw_response_path
- status
- error_message
serp_results
- id
- search_request_id
- result_type
- position
- title
- url
- domain
- description
- serp_feature
- parsed_at
result_type can distinguish organic results from ads, local results, images, videos, or featured snippets. serp_feature can record whether the result appeared inside a special module, such as a local pack or People Also Ask section.
This structure lets you answer questions like:
Which keywords lost visibility this week?
Which domains consistently appear for a target query?
Do results differ between New York and London?
Did a competitor begin appearing in paid results?
Which queries trigger local packs or video results?
Collect Results with Python
The example below shows the collection layer in a provider-agnostic way. The exact request format depends on the SERP API you use, but the surrounding logic remains similar.
python
import os
import time
import json
import requests
from pathlib import Path
RAW_DIR = Path("raw_responses")
RAW_DIR.mkdir(exist_ok=True)
API_URL = os.environ["SERP_API_URL"]
API_TOKEN = os.environ["SERP_API_TOKEN"]
def fetch_serp(
keyword: str,
country: str = "us",
city: str = None,
language: str = "en",
device: str = "desktop",
) -> dict:
payload = {
"q": keyword,
"gl": country,
"hl": language,
"device": device,
}
if city:
payload["location"] = city
response = requests.post(
API_URL,
headers={
"Authorization": f"Bearer {API_TOKEN}",
"Content-Type": "application/json",
},
json=payload,
timeout=30,
)
response.raise_for_status()
return response.json()
def save_raw_response(keyword: str, data: dict) -> Path:
timestamp = time.strftime("%Y%m%d-%H%M%S")
safe_keyword = "".join(
char if char.isalnum() else "_" for char in keyword.lower()
)
file_path = RAW_DIR / f"{safe_keyword}-{timestamp}.json"
file_path.write_text(json.dumps(data, ensure_ascii=False, indent=2))
return file_path
In production, avoid storing API credentials directly in code. Use environment variables, a secrets manager, or your deployment platform’s secret storage.
Parse the Results
A SERP response may contain many modules. Start with the data you will actually use; do not try to parse everything on day one.
For a basic rank tracker, organic results are usually the first priority:
python
def parse_organic_results(data: dict, search_request_id: int):
results = []
for item in data.get("organic_results", []):
results.append(
{
"search_request_id": search_request_id,
"result_type": "organic",
"position": item.get("position"),
"title": item.get("title"),
"url": item.get("url"),
"domain": extract_domain(item.get("url")),
"description": item.get("description"),
"serp_feature": None,
}
)
return results
def extract_domain(url: str) -> str | None:
if not url:
return None
from urllib.parse import urlparse
parsed = urlparse(url)
return parsed.netloc
If your provider returns a different schema, write a small adapter for it. Keeping parsing logic separate from collection logic makes it easier to switch providers or support multiple formats later.
Store Results in SQLite
SQLite is a good starting point for a small internal tool. It requires no separate database server and is easy to inspect locally.
python
import sqlite3
DB_PATH = "serp_monitor.db"
def init_db():
connection = sqlite3.connect(DB_PATH)
connection.execute(
"""
CREATE TABLE IF NOT EXISTS search_requests (
id INTEGER PRIMARY KEY AUTOINCREMENT,
keyword TEXT NOT NULL,
search_engine TEXT NOT NULL,
country TEXT,
city TEXT,
language TEXT,
device TEXT,
requested_at TEXT NOT NULL,
raw_response_path TEXT,
status TEXT NOT NULL,
error_message TEXT
)
"""
)
connection.execute(
"""
CREATE TABLE IF NOT EXISTS serp_results (
id INTEGER PRIMARY KEY AUTOINCREMENT,
search_request_id INTEGER NOT NULL,
result_type TEXT NOT NULL,
position INTEGER,
title TEXT,
url TEXT,
domain TEXT,
description TEXT,
serp_feature TEXT,
parsed_at TEXT NOT NULL,
FOREIGN KEY (search_request_id)
REFERENCES search_requests (id)
)
"""
)
connection.commit()
connection.close()
For a larger system, PostgreSQL is usually a better fit. It offers stronger concurrency, better tooling, and more options for analytical queries.
Run a Daily Job
You can start with a simple scheduled script. Once the volume grows, move to a task queue or a workflow engine such as Celery, Temporal, or Airflow.
python
from datetime import datetime, timezone
KEYWORDS = [
{"keyword": "project management software", "city": "New York"},
{"keyword": "best crm for startups", "city": "London"},
{"keyword": "email marketing tools", "city": "Singapore"},
]
def run_monitoring_job():
init_db()
for item in KEYWORDS:
try:
data = fetch_serp(
keyword=item["keyword"],
city=item["city"],
)
raw_path = save_raw_response(item["keyword"], data)
connection = sqlite3.connect(DB_PATH)
cursor = connection.cursor()
cursor.execute(
"""
INSERT INTO search_requests (
keyword, search_engine, country, city,
language, device, requested_at,
raw_response_path, status
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
item["keyword"],
"google",
"us",
item["city"],
"en",
"desktop",
datetime.now(timezone.utc).isoformat(),
str(raw_path),
"success",
),
)
request_id = cursor.lastrowid
parsed = parse_organic_results(data, request_id)
cursor.executemany(
"""
INSERT INTO serp_results (
search_request_id, result_type, position,
title, url, domain, description,
serp_feature, parsed_at
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
[
(
row["search_request_id"],
row["result_type"],
row["position"],
row["title"],
row["url"],
row["domain"],
row["description"],
row["serp_feature"],
datetime.now(timezone.utc).isoformat(),
)
for row in parsed
],
)
connection.commit()
connection.close()
except Exception as error:
print(f"Failed to collect {item['keyword']}: {error}")
if __name__ == "__main__":
run_monitoring_job()
This is intentionally simple. Once it works, add retry logic, request throttling, structured logging, and alerting.
Make Localized Comparisons Reliable
Localization is where many SERP monitoring projects become misleading.
If you compare a result collected in New York with one collected in London, you are not comparing the same search experience. Location, language, device, and even the time of collection can affect what the user sees.
Always store the full request context:
Keyword
Search engine
Country and city
Language
Device type
Request timestamp
API version or parser version
Then compare like with like. For example, compare “New York, English, desktop” results over time—not a mix of different locations and devices.
Add Alerts Without Creating Noise
A rank drop is not automatically an incident. Search results can fluctuate for many reasons, including personalization, regional variation, SERP layout changes, and normal ranking movement.
A useful alert should answer: what changed, by how much, and does it matter?
For example:
A target domain falls out of the top 10 for a priority keyword.
A competitor enters the top three for several related keywords.
A local pack appears where it previously did not.
A high-value page disappears from the first page for more than 24 hours.
The number of paid ads on a keyword increases sharply.
Avoid alerting on every one-position change. Instead, define thresholds based on business impact and keyword priority.
Use SERP Data in AI Workflows
SERP data can be useful input for AI systems, but it should not be treated as final truth.
A research agent might use search results to identify relevant sources, then visit the original pages, extract content, and generate an answer with citations. A RAG system can use search results to supplement an internal knowledge base with current public information.
A useful pattern is:
text
User question
|
v
Generate search queries
|
v
Retrieve SERP results
|
v
Select promising URLs
|
v
Fetch and extract page content
|
v
Cross-check and summarize
|
v
Answer with citations
Keep the source URL, title, snippet, retrieval timestamp, and search query with every answer. This makes the output easier to audit and helps users understand where the information came from.
Engineering Pitfalls to Avoid
A few issues tend to appear in almost every SERP monitoring project:
Mixing locations in one dataset. Store location and language as first-class fields.
Overwriting historical results. Keep a time-series record so you can analyze trends.
Parsing only the happy path. Search pages and API responses change; add validation and error handling.
Ignoring SERP features. Organic position alone may not explain visibility changes.
Requesting too aggressively. Use sensible schedules, caching, and rate limits.
Treating rankings as absolute truth. Use rankings as one signal alongside traffic, conversions, and business outcomes.
Collecting more data than needed. Define a clear purpose for every field and retention period.
A Good Starting Point
Start small. Choose 10 to 20 keywords that genuinely affect your business, monitor them in two or three priority markets, and store the raw responses.
Once you can reliably answer questions such as “Which pages changed this week?” or “Which markets show a different competitive landscape?”, expand the system. Add more keywords, more locations, additional SERP modules, and AI-driven analysis only when the basic pipeline is stable.
The most valuable SERP monitoring system is not the one that collects the most data. It is the one that helps your team notice meaningful change early enough to act.
來源:DEV Community · dev.to