HackerBooks 如何避免把負面提及當成書籍推薦

HackerBooks is a nice idea: point a pipeline at two decades of Hacker News comments and get a crowd-ranked bookshelf. It works well enough that the gaps are easy to miss. The gap that matters is the scoring. Every mention increments the same counter, so a book that commenters uniformly dislike can outrank a quiet classic.

By the end you will be able to pull HN comments, extract candidate book titles, resolve them to real works, and score them in a way that does not reward controversy.

Pull the comments

The HN search API is the practical starting point. Algolia mirrors every story and comment, and search_by_date lets you page forward with a timestamp cursor so you never re-fetch old data.

import requests, time

BASE = "https://hn.algolia.com/api/v1/search_by_date"

def fetch_comments(query, pause=0.2):
    cursor = 0
    while True:
        resp = requests.get(BASE, params={
            "tags": "comment",
            "query": query,
            "numericFilters": f"created_at_i>{cursor}",
            "hitsPerPage": 50,
        }, timeout=20)
        resp.raise_for_status()
        hits = resp.json()["hits"]
        if not hits:
            return
        for hit in hits:
            yield hit
        cursor = hits[-1]["created_at_i"]
        time.sleep(pause)

The cursor matters more than it looks. Offset-based pagination on a growing index skips entries when new comments arrive mid-crawl. A timestamp cursor is idempotent and lets you resume an interrupted run from a checkpoint.

Keep author, story_id, and created_at alongside the text. You need all three later, because the shape of the discussion is what separates a recommendation from an argument.

Find the titles

Commenters rarely use a consistent format. You get quotes, title case, and bare names. Two regexes cover most of it, then you filter the noise.

import re

QUOTED = re.compile(r'"([A-Z][^"]{2,79})"')
TITLE_CASE_RUN = re.compile(r"(?:[A-Z][a-z]+(?=\s|$)(?:\s+|,\s*)){2,5}")
STOP = {"the", "a", "an", "and", "of", "in", "to", "is", "it", "on"}

def candidates(text):
    found = set(QUOTED.findall(text))
    for match in TITLE_CASE_RUN.finditer(text):
        found.add(match.group(0).strip(" ,"))
    return {p for p in found
            if len(p.split()) >= 2
            and not STOP.issuperset(p.lower().split())}

This over-collects on purpose. Sentence fragments, product names, and people's names all survive. That is fine at this stage, because the next step discards anything that is not a real book. Filtering too aggressively here is the mistake that loses niche titles nobody quotes formally.

Resolve to a real work

This is the part the source hand-waves. "The Pragmatic Programmer", "Pragmatic Programmer", and "The Pragmatic Programmer: From Journeyman to Master" are one book. Matching on raw strings splits its score three ways.

import re

def work_key(title):
    t = title.lower()
    t = re.sub(r"[:\u2013\u2014-].*$", "", t)   # drop subtitles
    t = re.sub(r"\([^)]*\)", "", t)             # drop parentheticals
    t = re.sub(r"^(the|a|an)\s+", "", t)
    t = re.sub(r"[^a-z0-9 ]", "", t)
    return re.sub(r"\s+", " ", t).strip()

Then confirm each candidate against Open Library, which is free and needs no key.

def lookup(title, session):
    resp = session.get(
        "https://openlibrary.org/search.json",
        params={"title": title, "limit": 1}, timeout=15)
    docs = resp.json().get("docs", [])
    if not docs:
        return None
    doc = docs[0]
    return {"work": doc["key"], "title": doc["title"],
            "key": work_key(doc["title"])}

Confirming against a catalog does two jobs at once. It throws out the sentence fragments, and it gives you a stable work identifier so later editions and subtitle variants collapse into one row. Without that identifier, your leaderboard is really a leaderboard of title strings.

Score without confusing hate for love

The source states the problem plainly: "It counts mentions, so it's not perfect (praising a book and trashing it both count)." That is not a quirk. It is the expected behavior of any volume metric, and it gets worse on HN specifically, where a controversial book generates long arguments.

There is no single correct column. The honest fix is to combine them, weighting breadth over volume and using sentiment as a tie-breaker rather than a primary signal.

GOOD = {"love", "great", "excellent", "recommend", "favorite",
        "best", "must-read", "masterpiece", "worth"}
BAD = {"overrated", "boring", "waste", "disappointing",
       "hate", "avoid", "tedious", "dull", "abandon"}

def sentiment(text):
    words = set(re.findall(r"[a-z']+", text.lower()))
    return len(words & GOOD) - len(words & BAD)

def score(mentions):
    users = {m["author"] for m in mentions}
    threads = {m["story_id"] for m in mentions}
    liked = sum(1 for m in mentions if sentiment(m["comment_text"]) > 0)
    return len(users) * 3 + len(threads) * 2 + liked

Distinct users dominate the score because one person repeating a title in a thread says nothing about the community. Thread spread is next. Sentiment adds last, and only because a word list is genuinely bad at sarcasm. If a commenter writes "another masterpiece of self-help drivel", the lexicon reads it as praise.

Store the per-mention rows, not just the totals. When a book ranks oddly, you need to read the actual comments behind the number. An aggregate you cannot decompose is unverifiable.

What still breaks

Series names merge with individual volumes. "The Wheel of Time" and "The Eye of the World" are different rows to a catalog and the same thing to a commenter. You will double-count either way unless you add a manual override table for the handful of series that dominate the results.

Recommendation threads inflate everything they touch. A weekly "what are you reading" post generates dozens of low-information mentions. Consider scoring threads by their comment count or excluding recurring asks entirely.

Negation defeats the lexicon. "I did not enjoy it" scores neutral at best. If sentiment matters to your product, this is where you spend real effort, with a classifier rather than a word list.

Old books drift. A title discussed mostly in 2009 keeps its score forever. Decide whether your ranker is a lifetime reputation index or a current-interest feed, because it cannot be both.

Key takeaways

• Page the HN API with a timestamp cursor, not an offset, so growth during the crawl does not skip rows.
• Over-collect title candidates and let a catalog reject the noise; filtering early loses informal mentions.
• Collapse variants on a catalog work identifier, not a normalized string, or editions split your scores.
• Weight distinct users and distinct threads above raw counts, or controversy ranks as popularity.
• Keep every mention row so an odd ranking can be traced back to the comments that produced it.

Source

HackerBooks \u2013 HackerTube, but for books mentioned on HN

The source is a working site built largely by an LLM, covering roughly 16.5k books from 2007 onward. I added the pipeline code, the entity-resolution and scoring logic it does not describe, and the failure modes that come from counting mentions as approval.

Support this work

These write-ups are researched and published with no paywall, sponsor, or tracking. If one saved you an afternoon, a small tip keeps them coming.

USDT, USDC or USDD · TRC-20 (Tron)

TFTNsfyomKrnUutRjBTGVULp19ByW29KbY