SiteScope：免費技術 SEO 審核工具，按優先次序排列待修問題

For my college course I built SiteScope (vikramyadav.me), a free technical SEO audit tool. You enter a site, it crawls every page it can reach, and it tells you what to fix first. No sign-up needed to run an audit. If you want, you can create an account to have your sites re-audited daily or weekly.

The problem

Two things bugged me about free SEO checkers.

First, most test a single URL and hand back an unordered list. SiteScope follows internal links and the XML sitemap and checks every page it finds (up to 300 per audit). Then it ranks issues with one rule:

priority = severity × share of pages affected

A missing alt text on every page outranks a broken link on one page; a 5xx on half the site outranks both.

Second, a lot of sites load their content with JavaScript. SiteScope reads raw HTML, the way most AI crawlers do, so you can check whether your content needs JavaScript to show up. The tool itself uses no JavaScript framework. Every page, including the report, is server-rendered, because a tool that talks about JavaScript SEO should be readable by a crawler that runs none.

It also checks broken links, duplicate and near-duplicate pages (SimHash), the internal link graph (click depth, PageRank, orphan pages), titles, meta descriptions, H1s, canonicals, noindex and more. Reports download as PDF or JSON, and re-running an audit shows what was fixed and what is new since last time.

Tech stack

• FastAPI (Python 3.12) with Jinja2 templates; audits run as FastAPI background tasks
• PostgreSQL 16, where each finished result is one JSONB column
• Docker Compose with an Nginx reverse proxy and Certbot
• Azure VM: one small VM (2 vCPU, 1 GiB RAM) running two containers
• Cloudflare for DNS and proxying
• httpx + asyncio for crawling, BeautifulSoup for parsing, NetworkX for the link graph

What I learned

• A URL fetcher is an SSRF hole. If a stranger can type any URL, they will try the cloud metadata service or localhost. The guard refuses private, loopback and reserved addresses, checks every redirect hop, and connects to the exact IP it checked, so DNS rebinding can't swap the address in between.
• Don't trust a header anyone can send. My per-IP limit read Cloudflare's CF-Connecting-IP, but anyone hitting the server's IP directly could send that header with a new value each time. Now it only believes that header when the connection really comes from Cloudflare's IP ranges, and the limit lives in Postgres so a restart doesn't reset it.
• Keep dependencies small on a small VM. NetworkX's PageRank pulled in numpy and scipy, so I wrote my own PageRank and tested it against the reference. The scheduled re-audits run as a background task inside the app, with no cron, Celery or Redis.
• Severity alone gets the order wrong. Multiplying by the share of pages affected fixed it.
• An SEO tool shouldn't create thin pages. Report pages send noindex and are disallowed in robots.txt.
• DNS moves need care. Switching nameservers to Cloudflare replaces the whole zone, so I recreated the existing records first and issued SSL before turning on the proxy.

What's next

Some things are still open: score-drop emails are waiting on an SMTP provider, and external link checks stop at 200 links per audit.

Try it at vikramyadav.me and see the code on GitHub. Feedback welcome!