CrawlGuard：免費封鎖激進 AI 爬蟲並保護伺服器頻寬的工具
I built CrawlGuard: A free tool to block aggressive AI crawlers and protect server bandwidth
CrawlGuard 是一款免費工具，可協助網站管理者封鎖激進的 AI 爬蟲，保護伺服器頻寬。工具可產生 robots.txt、Apache .htaccess、Nginx .conf 及 Cloudflare WAF 規則，並提供即時測試器檢查網址路徑會否被允許或封鎖。工具在瀏覽器端執行，無需登入或帳戶。
Hey everyone! 👋
Over the last few months, I noticed a huge spike in aggressive web crawlers like Bytespider, GPTBot, and ClaudeBot hitting web servers. Unlike Google or Bing, many of these AI training scrapers ignore standard crawl delays and scrape content without sending any referral traffic back.
Manually looking up user-agents and writing complex rules for both robots.txt and reverse proxies was getting tedious, so I built CrawlGuard.
👉 Try CrawlGuard Free at crawlguard.dev
What it does:
- 1-Click Presets: Generate optimal rules (e.g., "Block All AI Training", "E-Commerce Protection", or "Balanced Modern Web").
-
Multi-Format Export: Spits out valid RFC 9309
robots.txt, Apache.htaccess, Nginx.conf, and Cloudflare WAF rules for bots that ignore robots directives. - Live Tester: Includes an interactive simulator at the bottom to test if your URL paths are allowed or blocked.
- 100% Free & Private: Runs client-side in your browser with no login or account required.
I’d love to hear your feedback on the UI, and if there are any specific malicious or rogue crawlers you'd like me to add to the registry!
來源：DEV Community · dev.to