跳到正文
DEV Community· ゴエンチョウ·· 15 天前AI 評分48

PageSift 1.2 加入 ITNA 存取障礙恢復引擎,並回顧 1.1 系列更新

PageSift 1.2: ITNA, an access-barrier recovery engine — plus the 1.1 roundup

AI 導讀

PageSift 1.2 加入 ITNA 存取障礙恢復引擎,可在 HTTP 擷取受阻時判斷是否改用瀏覽器重試,並整合 Cookie、出口連線及瀏覽器指紋管理工作階段。ITNA 預設關閉,按需啟用;不處理互動式 CAPTCHA,也不保證繞過所有反機器人系統。PageSift 全面開源,1.1 支援辨識 373 種程式語言。

正文

ゴエンチョウ

PageSift started as a Rust crawler chasing extreme performance. It has since grown into a web content engine — web scraping, document parsing, structured extraction, and an AI-agent surface — and 1.2 adds one more engine to it.

GitHub: https://github.com/YOLOW92/pagesift

1.2: ITNA, an access-barrier recovery engine

  • When an HTTP fetch gets blocked, it decides intelligently whether to escalate to a browser retry
  • TLS/HTTP2 browser fingerprinting over HTTP, HTTPS, and SOCKS5 proxies
  • A session identity pool: cookies, egress, and fingerprint managed together, isolated per user
  • Safe HTTP → browser handoff of session state
  • Tuned browser automation markers, with waiting for non-interactive JS challenges
  • A shared request budget that caps retries, timeouts, and resource use

ITNA is off by default and enabled on demand. It does not solve interactive CAPTCHAs, and it does not promise to bypass every anti-bot system.

Catching up on 1.1: documents and languages, massively expanded

I never got around to posting about the 1.1 series, so here it is in one go:

  • A long list of new file formats: legacy Office, ODF, WordPerfect, e-books, email, archives, databases, and more
  • 373 source-code languages recognized, with function/class-level structure extracted for mainstream languages
  • New bindings for Go, Java, C#, Ruby, PHP, Lua, Swift, Zig and more, plus LangChain and LlamaIndex integrations

Measured on 1.1:

  • PDF → Markdown: 2.4 ms/page
  • HTML → Markdown: 0.24 ms/page
  • Main-content extraction: 1.0 ms/page
  • Office parsing: 91.7 files/sec, about 3.40× xberg

1.1.1–1.1.3: still digging into the Rust layer

I implemented Zstandard compression/decompression in pure Rust — FSE, Huffman, LZ matching, cache-aware hash chains, all written from scratch.

Where it stands

PageSift has passed 200 stars on GitHub, and it is still developed and maintained by one person.

The goal was never just a fast crawler. I want it to be genuinely high-performance, low-overhead, self-hostable AI data infrastructure.

Fully open source — search YOLOW92 / PageSift on GitHub. Tests, issues, and technical discussion are all welcome.

來源:DEV Community · dev.to