如何構建簡易自我修復 Scrapy 智能體

A scraper can keep returning products after it stops collecting their prices. The requests succeed, the items arrive, and the crawl carries on. If you're scraping hundreds of thousands of products, finding that out at the end is expensive.

I've been building a small self-healing agent around my Scrapy projects to catch that problem and attempt a repair. In the video, I deliberately change the price class on my own test shop so the existing selector stops matching. The monitor detects the missing prices, closes the spider, and hands the failure to an agent.

The agent works on a copy of the project, captures fresh HTML, updates the selector and runs the tests. I can review its changes before allowing anything back into the live project.

The model is only one part of this. Most of the work is giving it a specific failure to investigate, a small area of code to change, and tests that catch a repair which breaks something that used to work.

The loop runs in this order:

1. Spidermon checks every tenth product and asks the spider to close when a price is missing.
2. The close action records the first offending URL, field and page object as a structured event.
3. Once the crawl has exited, the wrapper copies the project to a temporary directory.
4. scrapy savefixture captures the changed page in that copy.
5. The agent edits the page object and the new fixture, with up to three test attempts.
6. The controller reruns the checks independently and, if they pass, copies back only the selector file and the new fixture.

Catch the failure during the crawl

I'm using Spidermon to check that the products have prices.

Every product passes through a pipeline that records missing prices in Scrapy's statistics. After every tenth product, a signal-driven extension runs the Spidermon monitor suite against those statistics. If any product has a missing or blank price, the suite fails and its failure action requests that the spider close.

This is an item-count trigger, rather than Spidermon's time-based periodic scheduler. I want a check after each batch of ten products, not after a fixed number of seconds.

That gives me a failure while the crawl is still running, so I don't have to wait for a completed export to discover that the price column is empty.

You'll see some products arrive after the monitor fails. Scrapy is asynchronous, and requests already in progress or queued in the downloader can finish during shutdown. The repair wrapper waits for the crawl process to exit before it starts changing anything.

The rule here is deliberately blunt, since a missing price is enough to stop this crawl, and other projects will need different rules because an unavailable product might legitimately have no price. You need to decide what counts as a failure before asking an agent to fix it.

Give the agent a useful failure report

Rich handles the console output so I can follow the crawl, with the products, Spidermon's results and a finish message that gives the spider name, close reason, product count and elapsed time, but the repair process reads a structured event that the close action writes alongside it:

{
  "reason": "missing_product_price",
  "field": "price",
  "url": "https://coffee.hylnd7.com/labs/classic/product/midland-papua-new-guinea-bea016",
  "page_object": "base.pages.coffee_hylnd7_com.CoffeeProductPage"
}

The URL points at an offending product page and not whichever request happens to finish last during shutdown, because I keep the first failure event and don't let later responses replace it. That means the agent starts with a field, a URL and the class responsible for extracting it, and never has to work out what went wrong from an entire crawl log.

I also want to see what the agent does with that information, so its progress logs record fixture capture, model requests, file reads, edits, test attempts and the final result, with timings and file paths but without prompts, credentials or private reasoning.

Keep the extraction code easy to target

If navigation, extraction, validation and storage all live in one large spider, a repair agent has a lot of code to consider changing.

I use scrapy-poet to separate the extraction logic into page objects. The spider handles crawling. A page object extracts the fields for an item, and its URL pattern tells scrapy-poet which pages it handles.

For example, this is the relevant part of the original price field:

@field
def price(self) -> Optional[str]:
    value = self.response.css(
        "article.detail .detail-meta .price::text"
    ).get()
    if not value:
        return None
    return value.strip().lstrip("£$€").strip() or None

The products themselves are custom attrs items whose fields have corresponding extraction methods on the product page object, so the structure gives the agent somewhere specific to look, because the failure says price and a price() method controls how that value is extracted.

The write tool limits changes to the domain's page-object file and the new fixture's expected output, and the agent has no general-purpose shell and no tool that can edit the live project, settings, monitors or existing fixture expectations.

There's still a distinction between file-level and function-level scope, since the tool permits edits to the page-object module and not just the price() method, and tests help catch unwanted changes within that file. Here that breadth was needed, because currency() and currency_raw() read the same price element as price(), and the agent's repair updated all three methods in one pass.

Capture the new page through the project

When the crawl has stopped, the wrapper copies the project into a temporary directory, taking the source, settings, scripts, tests, existing fixtures and dependency lockfile and leaving out the virtual environment, caches, logs, exports and common secret files.

I chose a working copy because I want to inspect a proposed repair without changing the project I normally run.

The controller then runs scrapy savefixture in that copy:

scrapy savefixture \
  base.pages.coffee_hylnd7_com.CoffeeProductPage \
  "https://coffee.hylnd7.com/labs/classic/product/midland-papua-new-guinea-bea016"

This command comes with scrapy-poet, and it captures the page through the Scrapy project, using its downloader configuration, then saves the inputs needed to test the page object.

That's the process I use when first building the extraction logic, which is to capture a fixture, inspect the response, write selectors and test them, and the repair agent repeats it against the changed page.

The controller also cleans the saved HTML into readable HTML and plain text. Those versions help the agent understand the page, but it must develop selectors against the raw HTML, because cleaning can remove classes or change the structure, and a selector that matches the cleaned document might not match the actual response.

The agent's instructions tell it to stop without editing if the page really has no price, or turns out to be a block or error page, and not to invent a value to make the test pass.

Let the agent attempt a small repair

I'm using LangChain's Deep Agents, with GPT-6 Luna through OpenRouter.

I built the implementation with a coding agent, so I didn't write every line myself, but I wanted the process to stay easy enough to review. A fair amount of the code handles copying, subprocesses, validation and moving files back. The model's job is much smaller.

The repair sequence is:

1. Read the failure event and the fresh fixture.
2. Inspect the page-object code and raw HTML.
3. Find the element containing the missing field.
4. Update the selector and populate the new fixture's expected values from the page.
5. Run the price check and the full pytest suite.
6. Return a short explanation of the proposed repair.

The agent has read tools, a restricted file writer and a test tool. It can make up to three tool-driven validation attempts, within a step limit and a per-request timeout but with no overall time or cost budget. There are no subagents and no background repair service in this version.

For the demonstrated change, the price element's class had changed from .price to .nopriser-sre. The repair added the new class alongside the original selector:

value = self.response.css(
    "article.detail .detail-meta .price::text, "
    "article.detail .detail-meta .nopriser-sre::text"
).get()

Keeping the original selector matters because the older fixture still uses it, and a repair that works on the latest page but breaks previously supported pages shouldn't pass the tests.

Test the repair before copying it back

The new fixture gives us a regression test for the changed markup. The existing fixtures check that the repair hasn't broken the older layout.

I also want a stricter check on the price itself, which must be a nonblank, finite, nonnegative decimal string that agrees with the new fixture's expected price.

Without that check, the agent could write null into the expected output and get a passing comparison while leaving the original problem unfixed.

These checks still leave a gap, because the agent also fills in the new fixture's expected output, which means they confirm that the selector and the fixture agree with each other and not that the price is the right one.

After the agent finishes, the controller runs the checks again independently instead of trusting the model's claim that the tests passed, confirms that protected files haven't changed, and verifies that nobody edited the live project while the repair was running.

If validation succeeds, it can copy back the repaired page-object file and the new regression fixture, saving a backup and a diff and replacing the source file atomically, so the whole temporary project never goes over the live one.

Start with review-only mode

In the recording, I run:

uv run python scripts/crawl_and_repair.py --review-only

This starts the crawl, waits for a monitoring failure and runs the repair process, making real model calls and fetching a fresh fixture while leaving the live project unchanged.

I can then open the temporary project, inspect the selector change and read the test results. In the demo the whole process took about a minute, although that's a small example with one product page to investigate and not a general performance benchmark.

Running without --review-only allows the wrapper to apply a repair after validation:

uv run python scripts/crawl_and_repair.py

That starts a fresh crawl and repair session, so a review-only workspace is never promoted after the fact, and once a fix has been applied the wrapper leaves it to me to restart scraping.

What this version can and can't do

This example handles one failure type on one site, a missing product price caused by changed markup, and isn't a general repair system for every way a scraper can fail.

A nonblank price can still be wrong, since it might be an instalment amount, an old price or a value from the wrong product, and fixture tests only establish what we've told them to check while the agent helps populate the new expected output. A stronger check would compare the result with a price recorded before the markup changed, or confirm that the value appears as visible text in the raw HTML, and until something like that exists, reviewing the diff and the source page is still worth doing.

Keeping the old selector alongside the new one protects the older fixtures, but each future rename adds another alternative to it, so after a few repairs the selector will need tidying by hand.

The temporary copy isn't a security sandbox either. The file tools are restricted, but page objects and pytest execute Python on the host. This is a local development tool for a project I control, not something I'd give untrusted code and assume was isolated.

I could extend the failure rules or add separate repair agents for other problems later. For now, I want to keep the loop small enough that I can see why it stopped, what the agent changed and how the tests checked it.

The pieces I'd put in place first are the monitoring, the failure report, the modular extraction code and the tests. Once those exist, the agent has a much more specific job to do.

In the next video, I want to look at why discussions of self-healing scrapers so often focus on broken selectors, and what that leaves out.