Capterra 評論爬蟲將 8.6% 評論列的文字欄位錯讀為 $62，作者修正 RSC 解析

I run a visa agency in Bali. The half of my week that is not visas goes into Apify Actors, and one of them reads B2B software reviews — Capterra, Software Advice, GetApp, TrustRadius — and returns all four with the same field names.

Last night I was grepping my own output for junk. Not looking for anything in particular, just running a regex for field values that look like machinery rather than text. One hit came back often enough to stop me:

{
  "reviewId": "Capterra___7209002",
  "title": "Slack: Essential for remote work productivity...and employee engagement",
  "overallRating": 5,
  "pros": "$62",
  "cons": "Honestly, not much. The only time I get frustrated with it is when I need to invite partners (from external companies) into our workspace for targeted deal collaboration…",
  "comments": "$61"
}

cons is a real review. pros is the string $62. Not an empty field, not a truncation, not a price — a three-character pointer where 1,309 characters of someone's opinion of Slack should be.

How often, and to whom

I keep every run's dataset, so this is measurable rather than anecdotal. Across 16 runs and 409 Capterra review rows, 35 rows — 8.6% — carried a pointer in at least one text field: 34 in pros, 12 in comments, never in cons. On Software Advice, 5 pointer fields across 150 rows. On TrustRadius, 0 across 1,330.

The TrustRadius zero is the clue. TrustRadius goes through a completely different code path in my Actor — one that reassembles each review out of a slice of markup. Capterra and Software Advice share a path that reads structured objects out of the page. So this was not a site change. It was my parser, and it had been wrong since at least late September.

What $62 actually is

Capterra runs on Next.js App Router, which means the page does not ship a single JSON blob. It ships a React Server Components stream, pushed into the document in pieces:

<script>self.__next_f.push([1,"1:\"$Sreact.fragment\"\n3:I[339756,[\"/assets-bx-capterra/…"])</script>

Concatenate those pushes and you get a flat wire format of numbered rows. Mostly they are component references, but some are text:

61:T425,Slack has been part of the corporate toolkits of the companies I've worked for since 2017…

Row 61, type T, length 0x425 = 1,061 bytes, then the text. And in the review object that owns that text, React leaves this:

{"reviewId":"Capterra___7209002","overallRating":"5.0",
 "consText":"Honestly, not much. The only time I get frustrated with it is when…",
 "generalComments":"$61","prosText":"$62"}

That is React's reference syntax. A $-prefixed string in an RSC payload is a pointer to another row, not a value. The serializer outlines a long string — hoists it out of the object graph into its own row — and substitutes the pointer. The threshold in React's flight serializer is 1,024 characters.

I had read none of this. My parser pulled balanced JSON objects out of the stream, found "prosText": "$62", and dutifully wrote $62 into the dataset as the review.

Why a line-based reader misses the rows

The obvious fix is to build a lookup table of rows and resolve pointers against it. My first attempt did it by splitting the stream on newlines and matching ^([0-9a-f]+):T([0-9a-f]+),. It found row 61 and reported rows 62 and 63 as missing — even though 62:T51d, is right there in the HTML.

Here is the reason, lifted from the raw stream:

…companies should seriously consider adopting it.62:T51d,Since 2020, I've worked remotely as…

A text row declares its length, so it does not need a terminator — and it does not get one. The next row id begins immediately after the last byte of the previous row's text. On a line-based split, the only text rows you find are the ones that happen to follow a newline-terminated row. Every text row that follows another text row disappears, which is exactly the case when a page has several long reviews in a row.

So the rows have to be walked in order, and the length has to be honoured in bytes, not characters — T counts UTF-8 bytes, and review text is full of smart quotes and emoji:

_RSC_ROW = re.compile(rb'([0-9a-f]{1,8}):')
_RSC_TEXT = re.compile(rb'T([0-9a-f]+),')

def rsc_text_rows(blob: str) -> dict[str, str]:
    """Map every length-prefixed text row in the stream to the text it carries."""
    raw = blob.encode('utf-8')
    rows, position, size = {}, 0, len(raw)
    while position < size:
        head = _RSC_ROW.match(raw, position)
        if not head:
            newline = raw.find(b'\n', position)
            if newline < 0:
                break
            position = newline + 1
            continue
        body = head.end()
        text = _RSC_TEXT.match(raw, body)
        if not text:
            newline = raw.find(b'\n', body)
            position = size if newline < 0 else newline + 1
            continue
        start = text.end()
        end = min(start + int(text.group(1), 16), size)
        rows.setdefault(head.group(1).decode(), raw[start:end].decode('utf-8', 'ignore'))
        position = end + 1 if raw[end:end + 1] == b'\n' else end
    return rows

Then resolving is a recursive walk over the parsed object, replacing any string that is exactly $<row id> with that row's text. One detail worth stealing: React escapes a string that genuinely starts with $ by doubling it, so a reviewer who wrote $62 a month is fair arrives as $$62 a month is fair. That means a single-$ value is always a pointer and never literal text — and that a parser which ignores all of this also silently doubles the dollar sign on any review starting with a price. I had zero of those in 2,947 rows, so that half is read from React's source rather than measured, but it costs one line to handle:

if value.startswith('$$'):
    return value[1:]

The bias is the real damage

8.6% of rows losing a field sounds survivable. It isn't, because the loss is not random.

Here are the longest text values on one Capterra page for Slack — the 25 reviews on page 1 — split by whether React kept them inline or outlined them:

There is no overlap, and there can't be: the cut is React's 1,024-character rule. So the fields that went missing were precisely the three longest pieces of writing on the page. The 150-character "great tool, love it" reviews all came through perfectly.

If you are scraping reviews to feed a summarisation prompt, sentiment model or RAG index, that is the worst possible sample to lose. Short reviews are noise; the 2,000-character one from the IT executive explaining what Slack replaced and why is the entire reason to buy review data. My dataset was quietly dropping the signal and keeping the filler, and every run finished green with the expected row count.

That is the part I keep coming back to. The run succeeded. The row count was right. Nothing errored, nothing timed out, no site blocked anything. The only symptom was a two-byte string in one field of one row in twelve, and the only reason I found it is that I ran a regex looking for values that don't look like prose.

After

Same page, same 25 reviews, after the fix — and verified on the platform, not just on my laptop:

reviews: 25   pointer fields: 0
longest pros lengths: 1966, 1309, 926, 843, 805

Rows 1,966 and 1,309 are the two that used to be $63 and $62. The 926 was always fine. Both Actors that share this code path — Capterra & SaaS Reviews Scraper and its sibling — went out the same evening.

What to take from this if you scrape Next.js

• Any $-prefixed string value in an RSC payload is a pointer. If your scraper reads App Router pages and never resolves them, you are losing your longest strings right now and your row counts look fine.
• Length-prefixed rows are not newline-terminated. Walk the stream; don't split it.
• Add a junk canary to your validator. Mine is one regex over every string field: ^\$[0-9a-f]{1,8}$. Values that look like machinery — pointers, $undefined, bare [object Object], NaN — are cheap to detect and are the ones that pass every other check.
• Headless browsers don't have this problem. Read the rendered DOM and React has already resolved its own pointers for you. You pay a browser per page for it, which is why I parse the stream instead — but if you're scraping a handful of pages a day, that trade may not be yours.
• Check field-length distributions, not just fill rates. A fill-rate check would have given this a clean bill of health: pros was populated in 100% of rows. It was populated with $62.

The reviews were there the whole time, 1,061 bytes further down the same response. I just never followed the pointer.