career-ops 職位存活檢查器實測：誤判已關閉會令仍有效職位從後續搜尋中永久消失

career-ops is an open-source job-search pipeline: Node.js scripts driven by AI coding CLIs, with about 76k stars on GitHub. Before it adds a posting to your pipeline, scan --verify checks that the posting is still open. Where the job board has a public API, it asks the API. Otherwise it loads the page in a headless browser and reads it.

A "closed" or "blocked" verdict goes into your scan history for good, and that posting is skipped on every later scan. So the two ways to be wrong cost very different amounts. A wrong "open" costs one more check later. A wrong "closed" makes a live job disappear from your search without a trace.

Between 3 and 20 October 2026 I had 16 pull requests merged into career-ops, most of them in this liveness checker. This is what I found, how I measured it, and what the fixes look like.

How I found the bugs

The board was busy: 135 open PRs and 208 open issues on 11 October. The project's CONTRIBUTING lets bug fixes go in without an issue, so instead of competing for claimed issues I read modules nobody was working on. I started with the ATS keyword and CV skill checks, which are close to my NLP background, then moved to the liveness checker.

For the checker, I ran it on real postings and read every disagreement. The sources were public job-postings datasets (LinkedIn, USAJOBS) and live boards without an API. Several fixes also came out of the project's own conversations: a merge note from the maintainer, a follow-up the review bot offered, and three follow-ups from my reviews of other contributors' PRs.

A blocked tracker took the whole page down

Many people run a DNS filter that blocks ads and trackers: Pi-hole, AdGuard DNS, Control D, a hosts-file blocklist. Such a filter answers 0.0.0.0 for a tracker it blocks.

The browser check has a security guard that refuses requests to private network addresses, and 0.0.0.0 is on its list. Refusing the tracker is right. But the guard also marked the whole page "blocked", which scan --verify records permanently.

I found it while reviewing another contributor's PR that touches how the guard handles DNS answers. To measure it, I took 35 postings on 19 boards with no API, all live the day before, and checked each one with main's code and with the fix, behind two real filtering resolvers and with no filter as a control:

• On main, 12 postings behind AdGuard DNS and 20 behind Control D read "blocked".
• With the fix, none did. Behind Control D, 11 of those 20 read active with a visible Apply button.
• With no filter, both versions agreed on 32 of 35. The 3 differences were flaky page loads, in both directions.

The fix (#5015) leaves the guard's decision alone. The request is still refused, with the same error. The guard only tags the refusal when the only private addresses in the answer are the filter's:

  if (blocked.length > 0) {
    const denied = new Error(`Access denied: Egress guard blocked private target IP ${blocked[0].ip}`);
    if (blocked.every(({ hit }) => SINKHOLE_ADDRESSES.has(hit))) denied.livenessCode = 'dns_sinkhole';
    throw denied;
  }

A tagged refusal of a tracker, a script or an image no longer decides the page's verdict. Any other private address still counts, even next to 0.0.0.0. I rejected the simpler option, ignoring every private answer for subresources, because it would drop the guard's signal on a host that really resolves into a private network.

The follow-up, #5075, covers the posting's own host. When that host had no usable address, the guard reported "blocked". But no address is often about the run, not the posting: a DNS outage during the scan, or the user's own filter blocking the job board. Now it reads uncertain, and the posting is kept for another check. With the posting's own host failing DNS, 19 of 19 postings from 19 boards were dropped permanently on main, and 19 of 19 were kept with the fix.

The PR states the trade-off: a domain that is really gone now reads uncertain too, because one failed lookup can't tell a dead domain from an outage. A kept dead posting costs one more check later. A pinned live one never comes back.

One failed load failed the rest of the run

This one turned up while I was measuring another fix: a posting whose host no longer resolved broke the checks after it in the same run. I reproduced it on the code from before my DNS change, to rule that out, and traced it to the browser library the next day.

The checker runs every browser check in one tab. When a page fails to load (its host has no address, its certificate has expired, its connection is reset), the browser shows its error page a moment after the checker has moved on, and that interrupts the next page load. The interrupted page then finishes loading late and interrupts the one after it, often to the end of the run. Those checks fail as load errors, so scan --verify keeps the postings without having checked them, and the closed ones among them go into the user's pipeline. The root is a known Playwright issue (microsoft/playwright#35944, closed upstream without a fix).

Through the real CLI, on one list of 45 URLs with 6 failing loads spread among 39 postings that were live the day before:

• On main, 39 of 39 postings read as load errors, from the first URL to the last.
• With the fix, none did: 30 active, 1 expired (HTTP 404), 6 "no Apply control", 1 Cloudflare challenge and 1 timeout.

Over 36 failed loads, the browser's error page landed 1 to 69 ms after the failure was reported, median 4 ms. So the fix (#5299) waits for the browser's own signal instead of a fixed delay, for at most 2 s:

async function navigateSettled(page, navigate) {
  try {
    return await navigate();
  } catch (err) {
    const canWait = typeof page.waitForEvent === 'function' && typeof page.mainFrame === 'function';
    if (canWait && err?.name !== 'TimeoutError') {
      await page
        .waitForEvent('framenavigated', {
          predicate: (frame) => frame === page.mainFrame() && frame.url().startsWith('chrome-error:'),
          timeout: FAILED_NAVIGATION_COMMIT_MS,
        })
        .catch(() => {});
    }
    throw err;
  }
}

A timeout is left alone: a timed-out load shows no error page, and the next load cancels it, so waiting would only add 2 s. I rejected parking the tab on a blank page after a failure, which another part of the code does after a search, because that load is interrupted the same way. The review bot pointed out that the old page changing its own address could end the wait too early. I couldn't make that happen in real Chromium in 26 tries, but fixed it anyway, with a test the first version fails.

CI installs no browser, so the tests use a stand-in page that plays out the browser's order of events. The same sequences also run on real Chromium where it can launch.

A ground truth showed a case the issue had missed

I filed #4832 for live postings on seven non-English boards that read as having no Apply button. getManfred, a Spanish tech-job board, was one. Its Apply button says "¡Apúntame!" ("sign me up"), which the checker didn't know, so every live posting read "no Apply control".

getManfred's own feed marks every offer open or closed, so the A/B could measure against the board instead of against my reading of pages. The first run showed a case the issue had missed: one closed posting read open on both versions. Closed postings written in English often have an FAQ button such as "Can I apply if I live outside Spain?", and the English Apply pattern took it for an Apply button.

The page does say the offer is closed, but that sentence renders near the bottom, after the checker's first look on some pages. Above the title, from the start, a closed posting says when it closed ("Oferta cerrada el 27/07/2023"). Closing patterns are read before the Apply button, so an FAQ button can't outvote the date. The fix (#5350) is two patterns:

  /\boferta cerrada el \d{1,2}\/\d{1,2}\/\d{4}/i,  // a closing pattern, read before the Apply button
  /\bapuntame\b/i,                                // an Apply label, accent-free like the others

I left out the verb "aplicar": 73 of the 111 getManfred pages I measured have an FAQ button that uses it, closed ones included. On 136 real page loads:

• Live: all 28 open offers read active with the fix. On main, 25 read "no Apply control", and the other 3 read active only through their FAQ button.
• Closed: all 83 read closed with the fix. On main, 17 read active, all of them written in English and read through the FAQ button.

The board already knew the answer

No Fluff Jobs, a Polish tech-job board, was the last of my board families from #4832, and the checker read it backwards. A live posting puts its Apply button ("Aplikuj") inside a page header, which the checker skips as page chrome, so every live posting read "no Apply control". A closed posting moves to a search page with a promo link that also says "Aplikuj", so most closed postings read as open.

I'd planned label patterns. Then I found that the board's per-posting API says directly whether a posting is open. So the fix (#5428) is an API rung, as several other boards already have. The API answers "published" (open), "expired" (closed) or "does not exist". A status the rung doesn't know reads uncertain, never closed, and an answer it can't read falls back to the browser.

Through the real CLI on 102 postings (40 live from the board's search API, 59 closed from Wayback Machine archives, 1 gone and 2 made up):

• On main, all 40 live postings read "no Apply control", and 42 of the 59 closed ones read active.
• With the fix, 40 of 40 read active and 59 of 59 read closed, without opening a browser: 26 s against 348 s.

The review bot raised two valid points on it. I measured that they change no verdict on today's API answers and shipped them as a follow-up, #5533.

VDAB, the job board of Flanders' public employment service, looks like the second case. A closed vacancy there keeps its page and its Apply button, and only a banner says it's closed. Reading the page while it renders, main read 13 of 84 loads of closed vacancies as open. My open PR, #5703, asks VDAB's per-vacancy API, which the page itself uses. One choice in it is deliberate: a "pending" vacancy reads uncertain, not closed, because one pending vacancy had never been published at all, and a closed verdict is recorded for good.

How each fix was tested

Every PR adds tests that fail on main and pass with the fix. I checked each one with mutation variants: deliberately broken versions of the fix that the tests must catch. On #4810, 13 of 13 were caught; on #4833, 12 of 12. I also ran the full suite on each branch against main. Over three weeks it grew from about 11,600 to about 16,800 tests.

Among the things the maintainer's merge notes thanked were the edge-case tests: in the keyword check, the ones that keep "IT", "HR" and "AWS" from matching ordinary words; in the checker, the one for a button that is hidden or sits in the navigation.

The rest of the work

• Four fixes to the ATS keyword-coverage and CV skill-gap checks. The keyword check, for one, looked for a single English heading, so reports in the translated modes stopped with an error. Here is the same report under each mode's heading on main, from #4725:

  en  ## Keywords extracted         exit 0  coverage 80%
  de  ## Extrahierte Keywords       exit 1  No "## Keywords extracted" block found (or it is empty).
  es  ## Palabras clave extraídas   exit 1  same
  ko  ## 추출한 키워드              exit 1  same
  zh  ## ATS 关键词提取             exit 1  same

On the branch, all five exit 0 at 80%.

• One fix to the Gupy provider, whose API had moved to another host. It closed #4831, which I filed.
• Reviews of five other contributors' liveness and scan PRs, all merged. Four authors applied my findings or suggested tests before merge, and a fifth took my suggestion into a new version.
• #4832 is closed: every board in it has a merged fix. We split it by approach instead of racing. The contributor who wrote the providers for its Chinese boards moved them to per-posting APIs, another contributor took CareerViet, and I did the Dutch boards, getManfred and No Fluff Jobs.

Three more PRs are open: #5535 (HiringRoom's "Postularse" label, approved), #5691 (Tech in Asia's Apply link in the page header) and #5703 (VDAB's API). Since 9 October I've been on the project's contributors team. Thanks to the maintainer for quick merges and for merge notes that say exactly what they valued.

What I learned

• Mistakes aren't symmetric when a pipeline makes permanent decisions. A false "open" costs one more check later. A false "closed" can't be undone. Most of my liveness fixes move an unsure case from permanent to retry-next-time.
• Find a ground truth before trusting an A/B. getManfred's feed and VDAB's API measured the fixes, not my reading of pages. getManfred's feed also showed a case the issue had missed.
• Look for the board's own API before patching the page reader. No Fluff Jobs looked like a label-and-pattern job. Its API answered both sides directly, needed no browser, and left the rules every board shares untouched.
• Measure before changing a rule every board shares. The header skip looked like a bug on every board, but 821 real pages from 268 sites showed it hid only one board's Apply button. So #5691 (open) reads one narrow case, a link to the posting's own page, instead of every button in every header.
• Separate the change from the page. Live pages differ from one load to the next, so two runs can disagree for reasons that have nothing to do with the code. Running both versions on the same loaded page at the same moment settles it: on #5691's control pages, all 5 verdict differences came with identical button lists.
• Some bugs live between two systems. The failed-load chain came from the browser and its automation library disagreeing about when a failed load is over. Waiting for the browser's own signal stops the chain, at a cost of a few milliseconds per failed load.
• Reviews count as contributions. Reviewing another contributor's DNS change led straight to the bug #5015 fixes.

All 16 merged PRs: https://github.com/career-ops-hq/career-ops/pulls?q=is%3Apr+is%3Amerged+author%3Aroydonsequeira