無須逐頁捲動搜尋 40,000 名追蹤者名單:建置過程中的設計心得
Searching a 40,000-Follower List Without Scrolling: Design Notes From Building It
建置者分享一款 Instagram 追蹤者搜尋工具的設計取捨,透過資料供應商的搜尋功能查找名單,避免逐頁載入整個追蹤者清單。工具按帳戶、搜尋方向及詞語快取結果,並以免費額度、付費點數及多層防機械人措施控制成本與濫用。工具只查閲公開資料,搜尋結果可能不完整,尤其是大型帳戶。
Disclosure: I'm one of the people who built the tool linked in this post. I'm sharing the design thinking, not selling anything. Some AI assistance was used to draft this and I edited it.
A friend asked me a simple question last month: "Does this brand actually follow the creators it says it works with?" The brand had around 40,000 followers. Checking by hand meant scrolling for ages. That tiny annoyance turned into a side project, and building it taught me more about small tools than I expected.
Here's what I learned, in case you're building something similar.
The problem is a pagination problem
People assume "search followers" is a feature the platform just hands you. For your own account it is. For other accounts, you're usually stuck paging through results, and every page costs time and a request.
That makes a naive version terrible:
- User types a term.
- Your server walks every page of the follower list.
- You filter on the server.
For a 40k list that's many requests, per search, per user. It's slow and expensive, and it invites abuse.
What actually worked
1. Search at the source when you can
If your data provider exposes a search endpoint for followers or following, use it instead of downloading everything. You trade completeness for speed and cost. For this use case, "find one person" fits that trade well, which is why we built an instagram follower search around the provider's search instead of a full crawl.
2. Cache by the question, not the user
The same question gets asked repeatedly: same account, same direction (followers vs following), same term. Build the cache key from those three things.
import { createHash } from "node:crypto";
function searchKey(direction, account, term) {
const normalized = [direction, account, term]
.map((s) => s.trim().toLowerCase())
.join("|");
return createHash("md5").update(normalized).digest("hex");
}
// searchKey("followers", "somebrand", "alex")
MD5 is fine here because it's just a cache key, not a security feature. Normalizing first matters more than the hash: "Alex " and "alex" should hit the same entry.
3. Make "free" something you can afford
A daily free allowance gets people to try the tool; credits cover heavy use. The trick is that the allowance has to be tied to something you can enforce, such as a short lock after a free search, so a refresh doesn't reset it. I'm intentionally not sharing exact limits, because publishing them makes them easier to dodge.
4. Protect the endpoint from bots
Any free tool with an expensive backend call will be hammered. A few layers cost almost nothing:
- A hidden honeypot field in the form that real users never fill in.
- A challenge such as Cloudflare Turnstile on the submit action.
- A per-IP rate limit that's strict per minute and per day.
None of these is perfect alone. Together they remove most of the noise.
5. Say what it can't do
This was the most important product decision. The tool only works for public accounts. Private accounts stay private, and very large accounts may not return every entry. Saying that up front stops support emails and builds trust. If a tool claims to open private profiles, it's lying or after your credentials.
A quick user-side tip
Not everyone wants to search; some people just want to browse. We kept that as a separate page to view instagram followers and following lists, so each page does one job and ranks for its own intent instead of competing with the other.
Privacy and ethics notes
- Only public data is read. Nothing is bypassed.
- Treat results as personal data: don't build lists to resell or to harass people. In the EU, bulk collection can trigger GDPR duties.
- Don't ask users for their platform password. If a flow needs it for public data, something is wrong.
What I'd do differently
- Add an explicit "partial results" label when a list is very large, instead of burying it in the FAQ.
- Log search categories instead of search terms, to keep less personal data around.
- Write the unhappy-path copy (private account, no result, rate-limited) before the happy path.
Takeaways
- Cache by question, not by user.
- Prefer the provider's search over crawling everything.
- Layer cheap bot protection early.
- Be blunt about limits.
If you've built a small tool on top of someone else's API, what surprised you most? I'd love to hear in the comments.
來源:DEV Community · dev.to