Matek 如何用 RSS、LLM 評分器和 Node worker 搭建 Reddit 潛在客戶搜尋工具

Every time I shipped a side project, I lost evenings doing the same thing: searching Reddit for posts like "is there a tool that does X?" or "any alternative to Y?". Those posts are some of the warmest early leads a solo founder can get, but they are scattered across dozens of subreddits and fall off "new" within hours.

So I built a small system to do the finding for me. This post is about how it's put together, what broke along the way, and what it still can't do. No growth hacks, just architecture.

The shape of the system

There are two deploys and one database:

A Next.js app (App Router, Next 16, React 19) for the marketing site, the dashboard, billing and auth. Auth is Auth.js v5 with Google sign-in and email magic links.
A Node worker (plain ESM, no framework) running on a separate VPS under PM2. It's actually two processes: a keyword worker that does the polling and scoring, and a small HTTP search API the app calls for on-demand searches and validation.
MongoDB, shared by both.

The app never talks to Reddit directly. The dashboard writes workspaces (a product description, subreddits, keywords) to Mongo; the worker reads them, fetches posts, stores matches and scores them; the dashboard reads matches back. When the app needs something right now (validating a subreddit, triggering a backfill for a new workspace), it calls the worker's search API with a shared secret header.

Why split it? Next.js on a container is great at serving requests and terrible at long-running loops. Keeping the loops in a separate process means a deploy of the web app never interrupts a polling cycle, and a stuck Reddit request never ties up a web worker.

Reading Reddit: RSS, not JSON

The obvious route is Reddit's .json listings. In my setup those got 403s once traffic went through proxies, so the worker reads the public Atom feeds instead:

/r/{subreddit}/new/.rss for each watched subreddit
search .rss for keyword sweeps across Reddit

RSS gives you less than JSON (no scores, no comment counts), but for "has someone just asked for a thing like mine?" you mostly need title, body, author, link and timestamp, and Atom has all of those.

Requests go out through a proxy pool with a configurable number of "lanes", a minimum gap between requests and a bounded retry count on 403s. All of that is configured through env variables, which brings me to the first lesson.

The worker loops

worker/index.js is basically a handful of timers:

Poll every 30 minutes (configurable): fetch new for all watched subreddits, match against keywords, store new matches.
Search sweep every 6 hours: keyword searches across Reddit to catch posts outside the watched subreddits.
Scoring every 5 minutes: pick up unscored matches and send them to the LLM.
Backfill check every 30 seconds: when a user creates a workspace, they shouldn't stare at an empty dashboard for half an hour, so a backfill job pulls recent history (using a web search API) and seeds matches.
Cleanup every 24 hours, behind an explicit disable flag.

Splitting fetching from scoring was one of the better decisions. Fetching is cheap and bursty; scoring costs money and has rate limits. With separate loops, a slow LLM call never delays the next poll, and a Reddit outage never leaves the scorer idle with a backlog it can't reach.

Scoring with an LLM

Keyword matching alone is noisy. "Looking for a CRM" matches a buyer, but so does "I hate every CRM I've tried, rant incoming" and "I built a CRM, roast it". The scorer sends each match plus the workspace's product description to an OpenAI model and asks how likely it is that this post is someone looking for a solution like this product.

The interesting part isn't the prompt, it's the guardrails around it:

Hard caps everywhere. Max LLM calls per day globally, max scoring per user per day, a limit on how many items go through the first pass. A bug in a loop with an LLM call inside is a bug with a credit card attached.
Daily lead limits and alert thresholds. If a cycle produces way more matches than usual, that's usually a broken filter, not a sudden market, and I'd rather get a Discord alert than a thousand junk "leads".
Regression fixtures. There is a lead-quality.json fixture of real-world-style posts with expected outcomes, and every matching bug fix adds a case. It's the closest thing to a test suite for "is the filter still sane?".

Replies: drafted, never posted

For a lead worth answering, the app drafts a reply. The user edits it and posts it themselves. The tool never posts to Reddit on anyone's behalf.

That's a product decision, not a technical limitation. Reddit communities are very quick to spot bots, and I didn't want to build a spam machine that gets its users banned. Drafting saves time; posting stays human.

Lessons learned

1. Read env at call time, not at import. The worker is ESM, and imports are evaluated before the body of the entry file. If a module reads process.env.SOMETHING at the top level, it can run before the .env.local loader does, and you get undefined in production but not on your laptop. Now the env loader is the very first import, and modules read config inside functions (getWebshareKey(), pollExcludeUserIds()), with a test that fails if a value gets captured at import time.

2. Duplicated schemas will bite you. The app and the worker each have their own Mongoose schemas for the same collections. It keeps the worker deployable without the app's code, but every field change has to be made twice. I've left myself a loud note about it; a shared package would be the real fix.

3. Anything that deletes data gets a kill switch. The cleanup job exports old matches and then deletes them. It ships with a disable flag and refuses to delete if email isn't configured (so the export can't silently fail first). Boring, and exactly what you want at 2am.

4. Make one command the definition of "done". npm run verify runs a syntax check, the worker tests (including the lead-quality fixture and a mocked backfill pipeline test), next build and Playwright end-to-end tests. CI runs the same command on every PR. Solo projects need this more than team ones, because nobody else is going to catch it.

5. Build the web app as an image, not on the server. GitHub Actions builds a Docker image (Next standalone output, Node 22 Alpine, non-root) and the host just pulls it. One gotcha: NEXT_PUBLIC_* variables get inlined at build time, so changing them in the hosting dashboard does nothing until you rebuild.

Limitations (honestly)

30-minute polling isn't real-time. On fast subreddits, someone may answer before you see the post. You can poll more often, but you pay for it in proxy traffic and rate limits.
RSS is a thin view of Reddit. No vote counts, no comment trees, and feeds only show recent items, so if the worker is down for long, it misses things that backfill has to recover.
The LLM is a ranking aid, not an oracle. It's good at pushing "is there a tool for…" above rants and launches. It still produces false positives, and the product description you give it matters a lot.
Dependence on Reddit's public endpoints. If those change, the fetch layer changes with them.

Wrapping up

If you want to build something similar, the core is small: an RSS fetcher, a keyword matcher, a queue of unscored items and an LLM call with strict caps. The rest is making it boring and reliable.

I turned this into a product called IndiePilot (there's a free plan) for founders who'd rather not run the plumbing themselves. Either way, I'm happy to answer questions about the architecture in the comments: what would you do differently?