跳到正文
DEV Community· Sergio·· 12 天前AI 評分49

用 35 行 JSON 將 Google Search 轉換成 API

Turning Google Search into an API with 35 Lines of JSON

AI 導讀

Relinos 可透過可編程瀏覽器,將 Google Search 的搜尋結果擷取為結構化 JSON,包含標題、網址、摘要、AI 生成答案及相關搜尋。它支援瀏覽器互動、分頁、快取及 XPath 資料擷取,所得 JSON 可供 LLM 或自動化流程使用。

正文

Google Search results contain titles, URLs, snippets, AI-generated answers, and related searches.

Here's how to extract them as structured JSON using Relinos, a programmable browser.

{
    "url": "https://www.google.com",
    "render": true,
    "actions": [
        ["actionWaitForAndSelect", "//textarea[@name='q']", "1"],
        ["actionType","your search term",""],
        ["actionSleep", "1", ""],
        ["actionKeyboardEnter", "", ""],
        ["actionSleep", "2", ""]
    ],
    "pagination": ["//*[@id='pnnext']", "2"],
    "cache": "1d",
    "format": "json",
    "extraction": {
    "query": { "$xpath": "//*[@id='ti6dpd']" },
    "result_count": {
      "$xpath": "//div[@class='XhWQv qd6zO ApHyTb']",
      "$extract": "\\D*(\\d[,\\d]*) .*"
    },
    "ai": {
      "$xpath": "//*[@class='mZJni Dn7Fzd']", "$format": "md"
    },
    "results": {
      "$xpath": "//div[contains(@class, 'MjjYud')]",
      "$list": true,
      "title": { "$xpath": ".//h3[contains(@class, 'LC20lb MBeuO DKV0Md')]" },
      "url": { "$xpath": ".//a[contains(@class, 'zReHs')]/@href" },
      "site": { "$xpath": ".//span[contains(@class, 'VuuXrf')]" },
      "snippet": { "$xpath": ".//div[contains(@data-snf, 'nke7rc')]" }
    },
    "related_searches": {
      "$xpath": "//*[@class='qR29te']", "$list": true
    }
  }
}

Those lines is all it takes, Here a shortened example output for the search term "Elon Musk"

{
    "query": "Elon Musk",
    "result_count": "157,000,000",
    "ai": null,
    "results": [
        {
            "title": "Elon Musk (@elonmusk) / X",
            "url": "https://x.com/elonmusk",
            "site": "X · elonmusk",
            "snippet": "The more I read the news, the more I think that becoming multiplanetary 
            is a necessity. I hope we will have enough time to prepare for life on the Moon and, 
            ...Read more"
        },
        {
            "title": "Elon Musk",
            "url": "https://en.wikipedia.org/wiki/Elon_Musk",
            "site": "Wikipedia",
            "snippet": "40 minutes ago — Elon Reeve Musk is an industrialist, technologist, and 
            former public official who is the chief executive officer (CEO) and largest shareholder 
            of Tesla and ...Read more"
        },
    ],
    "related_searches": [
        "Elon Musk book", "Tesla", "The Boring Company",
    ]
}

Automating Browser Interactions

Many modern websites generate content dynamically using JavaScript, which means a simple HTTP request will not return all the data we're looking for. Relinos supports both simple HTTP requests and browser rendering through Chromium via the render key.

But when we render sites we often want to take actions that a normal user would make like clicking, typing or scrolling a site. In Relinos these interactions are simply a sequential list. actionWaitForAndSelect for example waits and selects the specified element, actionClick later clicks the prevously selected element.

This is an intentional limitation. By keeping actions linear and leaving complex logic to the calling application, routines remain predictable, easy to understand, and straightforward to debug. Combined with Relinos's debug mode, this makes identifying and fixing issues much easier.

"actions" = [
    ["actionWaitForAndSelect", "//textarea[@name='q']", "1"],
    ["actionType","your search term",""],
    ["actionSleep", "1", ""],
    ["actionKeyboardEnter", "", ""],
    ["actionSleep", "2", ""]
]

Pagination often requires writing additional code to navigate between pages and combine their results. Relinos handles this through the pagination key containing the next-page selector and the total number of pages to visit.

"pagination": ["//*[@id='pnnext']", "2"]

Relinos follows the next-page selector, applies the same extraction to each page, and combines the results into a single JSON response.

During development, we often execute the same request repeatedly. With "cache": "1d", Relinos can reuse previously collected results for one day or any duration instead of loading the website again. This makes experimentation faster and reduces unnecessary requests.

Extracting Structured Data with XPathSchema

We can get data in "html", "md" or "json" format. We can build a json structure using the extraction key.
We could for example reason that google results consists of a list of search result each containing a title a url a description. We can use "$xpath" on each field to describe how we can locate the element in the full html or starting at its parent element.

"results": {
      "$xpath": "//div[contains(@class, 'MjjYud')]",
      "$list": true,
      "title": { "$xpath": ".//h3[contains(@class, 'LC20lb MBeuO DKV0Md')]" },
      "url": { "$xpath": ".//a[contains(@class, 'zReHs')]/@href" },
      "site": { "$xpath": ".//span[contains(@class, 'VuuXrf')]" },
      "snippet": { "$xpath": ".//div[contains(@data-snf, 'nke7rc')]" }
}

In the example, results uses an absolute selector, all children of result use a relative selector.
Since both html and json represent trees this format is very expressive and allows to create complex but readable automations fast.

Website structures change, and selectors eventually break. But because we've separated the data structure from its selectors, fixing an extraction usually means updating a few XPath expressions rather than rewriting the entire scraper.
Xpathschema also has other special keys next to "$xpath", there is more info on the repository

In the end we get our google search results back

From One Website to Many

We could take this further by defining similar routines for DuckDuckGo, Bing, YouTube, and other websites, then parameterizing them to make them even easier to reuse:

{
    "name": "google_search",
    "params": {
        "search_term": "Cute cat pictures"
    }
}

Combining their results would give us the foundations of a metasearch engine, similar to SearXNG, accessible through a single API.

Furthermore, because the output is structured JSON, these routines can feed directly into LLMs or automated pipelines.

The web contains countless useful interfaces. Relinos makes them programmable.

If you want to try Relinos yourself you can download it on https://github.com/realsirjoe/relinos

來源:DEV Community · dev.to