跳到正文
DEV Community· Sukarth·· 3 小時前AI 評分59

AI Studio Exporter 以網站內部 RPC API 取代 DOM 擷取,將 200 則對話匯出由約 150 秒縮至不足 1 秒

Going from 150s to <1s: Replacing DOM scraping with a site's internal RPC API in a Chrome extension

AI 導讀

作者開發的開源 Chrome 擴充功能 AI Studio Exporter 改用 Google AI Studio 前端使用的內部 API,將 200 則對話的匯出時間由約 150 秒縮短至不足 1 秒。

正文

Google AI Studio has no way to export a conversation. I wanted one, so I built an open source Chrome extension, AI Studio Exporter. Version 1 did what every exporter does: scroll the page to the top, wait for messages to render, and read them out of the DOM. Version 2 stopped touching the page at all and calls the same internal API that the AI Studio frontend calls.

For a 200-message conversation, exports went from about 150 seconds of automated scrolling to under a second. This post is about how it is put together, and the parts that were harder than they looked.

Why scraping was the wrong tool

Three things were wrong with reading the page:

  • It is slow. AI Studio renders only the turns near what you are looking at, so a scraper has to scroll each one into view and wait for it. That wait is most of the 150 seconds.
  • It misses things. Anything the page has not drawn, or drew late, is gone. Long chats lost messages in the middle. Fluctuating network conditions or a slow machine could worsen this, making the page drop messages without the scraper knowing.
  • It cannot see the files. An attachment on the page is a small preview image. The export contained that thumbnail, not the PDF you uploaded. Several of the bug reports filed against version 1 were about exactly this.

The page is a view of data that already exists on a server. The sensible move was to ask the server.

Finding the API

Open the Network tab while AI Studio loads a conversation and you can see the requests the frontend makes. Three methods matter, and they are all the extension calls:

  • ListPrompts, which returns your library
  • ResolveDriveResource, which returns one conversation
  • GenerateAccessToken, which returns a token for Google Drive

The responses are not JSON objects. They are protobuf-shaped nested arrays: every field is a position, and unset fields are null. The part of the codebase that knows the wire format is a decoder with a table of indexes, built by comparing responses with what the UI showed.

That decoder taught me one lesson twice. A prompt showing "Temperature 1" in the UI had 1 at index 0 and 0.95 at index 4, so index 4 is top-P, not temperature, and the two values being close to each other makes them easy to confuse. Likewise, something I assumed marked reasoning blocks actually marked answers: every export mislabelled thinking as the answer until I checked it against real data.

Because this API is undocumented and Google can change a field at any time, every read goes through tolerant helpers. A shape change degrades to missing metadata instead of a thrown exception, which is always a good development practice when you are reading data from a source you do not control.

The CORS problem and the MAIN world

The endpoint replies with Access-Control-Allow-Origin: https://aistudio.google.com. A fetch from an extension's content script runs in the isolated world and carries the extension's origin, so the browser rejects it. The request has to come from the page itself.

The solution is a small script that runs in the page's MAIN world at document_start, declared in the manifest:

{
  "matches": ["https://aistudio.google.com/*"],
  "js": ["src/api/rpc.js"],
  "world": "MAIN",
  "run_at": "document_start"
}

The rest of the extension runs in the isolated world and talks to it over window.postMessage, with a tag and a request id, and the MAIN-world script only answers messages that carry the right tag and only calls an allowlisted set of methods. It does nothing until asked, and it is the only thing in the main world.

Running there also gives access to document.cookie, which matters for authentication.

Authentication

Google's web endpoints use a scheme called SAPISIDHASH. The page already has a SAPISID cookie; the header is derived from it:

// SHA-1 over "<unixSeconds> <SAPISID> <origin>"
const seconds = Math.floor(Date.now() / 1000);
const hash = await sha1Hex(`${seconds} ${sapisid} ${origin}`);
const header = `SAPISIDHASH ${seconds}_${hash}`;

That header, an API key found on the page and the account index go with each call. Drive is stricter: it rejects SAPISIDHASH and wants OAuth, so the extension asks GenerateAccessToken for a bearer token and uses that for Drive.

Getting the real attachments

In the conversation data an attachment is a bare Drive id. With a token you can ask Drive for the file itself, which is why the export now holds your PDF and not a picture of its first page.

Files created in Google's own editors cannot be downloaded as they are, so they are exported through Drive's export endpoint:

Google file Saved as
Doc Markdown
Sheet CSV
Slides PDF
Drawing PNG

Anything else Google-native falls back to a PDF. Files over 100 MB are refused rather than buffered, so one enormous file cannot exhaust the tab's memory.

The fallback, and keeping it honest

In the end, I decided to also retain the DOM scraper, so that if the API path fails, the extension still works to a certain extent: the export falls back to reading the page, and the result records which route produced it (source: "api" or "dom"). The toolbar badge shows an amber mark when that happened, and the file is missing everything the page cannot show: model settings, token counts, timestamps and the real files.

A fallback you never run is a fallback that does not work, so there is a browser test for it: Playwright launches a real Chrome with the real extension, fulfils the AI Studio page with a fixture and answers every API call with a 500, then checks that an export still downloads with the messages, the reasoning and the image in it.

What 137 passing tests missed

I had a large suite of unit tests, all green, and then ran an export on a real 226-message conversation with 42 attachments. It found a bug none of them could. When two attachments have the same name, the exporter writes the second as name (2).png. The Markdown link to it ended at the first ), because encodeURI does not escape parentheses, so 12 of the 42 links were dead.

No test fixture had two attachments with the same name, and real conversations do this all the time. Tests that you invent only contain the shapes you thought of. Build them from real data.

Notes on Manifest V3 (MV3)

  • Scheduled backups. The extension can back up a whole library on a schedule. In Manifest V3 that means chrome.alarms and a service worker that may have been evicted, so the alarm handler has to rebuild all its state from storage every time. The actual export cannot run in the service worker, because the API needs the page's origin and building an archive needs a DOM, so the alarm opens an AI Studio tab in the background, asks the content script there to run an incremental export, and closes the tab afterwards.
  • Permissions. The final extension asks only for storage, alarms and host access to AI Studio, plus one host for the Pro licence check. Downloads use a blob URL and an anchor with a download attribute in the content script, which needs no downloads permission. I removed activeTab, scripting and downloads from the V1 extension because nothing uses them in V2, and the Chrome Web Store listing went through.
  • Windows paths. A library archive that opened fine in WinRAR was rejected by Windows Explorer as invalid. I figured out that the issue was with the length of the file paths. One attachment had a 218-character name, which put a path at 260 characters. Names inside archives are now capped to prevent this issue.

A few hard truths

  • The extension uses an undocumented API. It will change, and when it does the export degrades to the page-scraping route until I fix/update the extension to work with the change.
  • I do not know how Google reads its terms about a tool like this, and neither does anyone else who has built one. It makes the same requests your browser already makes while you use AI Studio, from your own signed-in session. Chrome Web Store review checks the store's own policies, not this question, so I treat it as an open risk and not as clearance. Other extensions that read a site's own API exist and are widely used, but that is not the same as permission.
  • The paid tier is a one-time licence for whole-library export, scheduled backups and six extra formats. The check is client-side JavaScript in an open source extension, so anyone can remove it. I didn't obfuscate it, since it would not stop a determined user from removing it, and would just hurt the code's maintenance and auditability. All the code, including the Pro features, is MIT licensed.

One last thing: if you happen to have maintained something built on a private API, I would love to know: how do you notice the day the API changes? Is it through a user report, or do you have a monitoring script that checks the endpoint every day?

Thanks to everyone who read this far. I hope this post helps anyone who wants to build a Chrome extension that calls a site's own internal API instead of scraping the page. Be sure to check out the relevant links below, and other user friendly guides at my site: exportaistudio.vercel.app/guides.

Links

來源:DEV Community · dev.to