以 Tranco 和 Chrome UX Report 免費數據為 1,700 款 AI 工具排名，取代 SimilarWeb

I run an AI tools directory with about 1,700 tools. Visitors want two things from it: which tools are actually popular, and which ones are taking off right now. For a while I scraped SimilarWeb for that. Then SimilarWeb started blocking my server's IP, and paying for their API wasn't an option for a side project.

It turns out you can get surprisingly far with two free, public lists. This post shows the whole monthly job — about 100 lines of Node — and the traps I hit.

The two data sources

Tranco top 1M (tranco-list.eu) is a research ranking of the top million domains. It combines several popularity sources and averages them over 30 days, which makes it far harder to game than any single list. You get an exact rank (1,openai.com-style CSV in a zip), updated daily.

CrUX top lists (the zakird/crux-top-lists repo on GitHub) are derived from the Chrome UX Report. They don't give exact ranks, only buckets: top 1k, 5k, 10k, 50k, 100k, 500k, 1M. They're keyed by origin (https://www.example.com), updated monthly.

Tranco is the primary signal. CrUX fills a gap: sites that Chrome users visit a lot but that aren't in Tranco's top million (often newer tools with a burst of real users).

Loading both lists

const TRANCO_URL = 'https://tranco-list.eu/top-1m.csv.zip';
const CRUX_URL =
  'https://raw.githubusercontent.com/zakird/crux-top-lists/main/data/global/current.csv.gz';

// domain → exact rank
export async function loadTranco() {
  const zipPath = path.join(os.tmpdir(), `tranco-${process.pid}.zip`);
  fs.writeFileSync(zipPath, await download(TRANCO_URL));
  try {
    const csv = execFileSync('unzip', ['-p', zipPath], { maxBuffer: 64 * 1024 * 1024 }).toString();
    const ranks = new Map();
    for (const line of csv.split('\n')) {
      const i = line.indexOf(',');
      if (i > 0) ranks.set(line.slice(i + 1).trim().toLowerCase(), Number(line.slice(0, i)));
    }
    return ranks;
  } finally {
    fs.unlinkSync(zipPath);
  }
}

// host (without www.) → bucket (1000 / 5000 / … / 1000000)
export async function loadCrux() {
  const csv = zlib.gunzipSync(await download(CRUX_URL)).toString();
  const tiers = new Map();
  for (const line of csv.split('\n').slice(1)) {
    const [origin, rank] = line.split(',');
    if (!origin || !rank) continue;
    let host;
    try {
      host = new URL(origin).hostname.toLowerCase().replace(/^www\./, '');
    } catch {
      continue;
    }
    const r = Number(rank);
    // http:// and https://, www and non-www collapse to one host: keep the best bucket
    if (!tiers.has(host) || r < tiers.get(host)) tiers.set(host, r);
  }
  return tiers;
}

Two details matter here. CrUX lists http:// and https:// origins, with and without www., separately — after normalizing to a bare host you get duplicates, so keep the best bucket. And Tranco is about 1M lines; a Map handles it fine, no database needed.

I also refuse to run the update if a download looks truncated:

if (tranco.size < 900000 || crux.size < 500000) throw new Error('lists incomplete, skipping update');

A half-downloaded list would silently mark hundreds of tools as "unranked" and wreck the month-over-month comparison.

Matching tools to domains — the subpage trap

The naive approach is "take the tool's URL, look up its hostname". That breaks on products that live on a big company's domain: if a tool's URL is openai.com/sora, the hostname lookup gives it OpenAI's rank, and suddenly a single product sits next to ChatGPT.

So a tool only gets a rank if its URL is the site's root (with a few root-like paths allowed):

const ROOT_PATH = /^(\/?|\/(en|en-us|us|home|index\.html|app))\/?$/i;

function parseToolUrl(url) {
  try {
    const u = new URL(/^https?:\/\//i.test(url.trim()) ? url.trim() : `https://${url.trim()}`);
    return { host: u.hostname.toLowerCase().replace(/^www\./, ''), isRoot: ROOT_PATH.test(u.pathname) };
  } catch {
    return null;
  }
}

Then the decision per tool is simple: exact Tranco rank if present, otherwise the CrUX bucket, otherwise nothing.

if (parsed?.isRoot) {
  if (tranco.has(parsed.host)) {
    globalRank = tranco.get(parsed.host);
    tier = tierOf(globalRank);          // keep a bucket for every ranked tool
    source = 'tranco';
  } else if (crux.has(parsed.host)) {
    tier = crux.get(parsed.host);
    source = 'crux';
  }
}

Category pages then sort by COALESCE(global_rank, rank_tier) ASC NULLS LAST: a CrUX-only tool in the top-50k bucket is treated as rank 50,000, so it lands just after the Tranco tools ranked better than that — a conservative spot, since all we know is "somewhere in the top 50k". Unranked tools go last.

This month's run:

Tranco 1000000 domains, CrUX 985563 domains
Updated 1743 tools: Tranco rank 1214, CrUX bucket 54, unranked 475 (of which subpage URLs 53)

So ~70% of the tools have an exact rank, CrUX adds 54 more, and 53 tools are deliberately left unranked because they're a subpage of a bigger site.

Month-over-month history

The rank itself goes into the tool table. In addition, every run writes one row per tool and month into a history table — including the host the rank was measured on:

INSERT INTO tool_rank_history (tool_id, month, tranco_rank, list_id, host)
SELECT u.id, date_trunc('month', NOW())::date, u.global_rank, 'update-rankings', u.host
FROM unnest($1::text[], $2::int[], $3::text[]) AS u(id, global_rank, host)
WHERE u.global_rank IS NOT NULL
ON CONFLICT (tool_id, month) DO UPDATE
  SET tranco_rank = EXCLUDED.tranco_rank, host = EXCLUDED.host;

unnest over arrays turns 1,700 updates into one statement, and ON CONFLICT makes the job safe to re-run in the same month.

Why store the host? Because tools rebrand and move domains all the time. When a tool moves from sometool.app (rank 800,000) to sometool.com (rank 90,000), that's not growth, it's a different domain. Without the host column, every domain move in the directory would show up as a spectacular "riser".

"Fastest growing" without the noise

The growth query compares the last two months, same tool, same host:

SELECT t.slug, t.name, n.tranco_rank AS current_rank, p.tranco_rank AS previous_rank
FROM tool t
JOIN tool_rank_history n ON n.tool_id = t.id AND n.month = $1
JOIN tool_rank_history p ON p.tool_id = t.id AND p.month = $2 AND p.host = n.host
WHERE n.tranco_rank <= 500000                       -- $3
  AND p.tranco_rank >= n.tranco_rank * 1.15         -- $4
ORDER BY p.tranco_rank::float / n.tranco_rank DESC;

Three choices in there, each learned the hard way:

1. Ratio, not absolute places. Climbing 2,000 places at rank 400,000 is noise; climbing 2,000 places at rank 5,000 is huge. Sorting by previous / current treats both fairly. The displayed percentage is (previous - current) / previous, so the 1.15 threshold means "climbed at least 13%".
2. A rank cap. Below ~500k, Tranco ranks jump around a lot month to month even for sites with stable traffic. Capping the current rank at 500k removed most of the random entries.
3. Dedup by host afterwards. A directory ends up with two entries on one domain now and then (a web app and a separate product page for its "pro" variant). The page keeps the first entry per host.

Tools that entered the top 1M this month (no row last month, but the tool was already in the directory) go into a separate "newcomers" list instead of getting an infinite growth ratio.

This month 91 tools cleared the bar. The top of the list: an AI character chat platform went from #274,746 to #128,591 (+53%), an AI study app from #120,721 to #57,225 (+53%), and a multi-model chat app from #117,931 to #58,380 (+50%). Study tools and "many models in one app" products each took two of the top 10 spots.

Caveats

• Rank is not traffic. Tranco tells you a site is more popular than another, not how many visits it gets. I show ranks, never visitor estimates.
• Domain-level only. Anything on a shared domain (*.vercel.app, Hugging Face Spaces, a product page on a big company site) can't be ranked this way. That's why 475 of 1,743 tools have no rank; the directory still lists them, sorted after ranked tools.
• Monthly granularity. Tranco updates daily, but a monthly snapshot is plenty for a directory and keeps the history table tiny.

The whole job

One cron entry on the 3rd of each month (paths simplified):

0 5 3 * * cd /srv/app && node scripts/update-rankings.mjs >> logs/update-rankings-cron.log 2>&1

Download two lists, normalize hosts, one bulk UPDATE, one bulk INSERT … ON CONFLICT, and the pages read from the database with a one-hour cache. It costs nothing, needs no API keys, and unlike the scraper it replaced, nobody can block it.

The live result of this month's run is the fastest-growing AI tools list on NavifyAI.com — happy to answer questions about the data in the comments.