四份履歷連續獲 100 分,CV by JD 如何重建評分器
Our resume scorer said 100/100 four times in a row. Here's how we rebuilt it
CV by JD 創辦人因四份針對不同職位的履歷連續獲 100 分而重建評分器,改由生成式 LLM 抽取職位要求、JEV 分類器逐項判斷,再以程式計算總分。系統會核對要求中的引文是否出現在職位描述中,評分與回饋亦使用同一組判斷。測試中,12 份忠實履歷全部通過,19 項刻意植入的虛構內容全部被檢出;JEV 每次呼叫約 $0.0002,處理 12 至 18 項要求約需 450 ms。
A user emailed us in September with a short, polite complaint. Their first two resumes scored 71 and 62. Then they tailored four more resumes to four different job descriptions, and every one scored 100 out of 100. None of them came back with a single suggestion.
I'm the founder of CV by JD. We take your resume and a job description, rewrite the resume for that job, and give you an ATS-style match score (ATS is the applicant tracking software employers use to filter applications) with a list of what's still missing. That email turned out to be one of the most useful bug reports we've had, and this post is about what it led to.
Where the 100s came from
No model produced those scores. When the backend returned no analysis, the frontend quietly fell back to an old client-side heuristic. It gave points for having an email address, a LinkedIn URL, the word "experience", more than 500 words and an action verb. The points added up to 120 and were capped at 100, so every normal resume scored perfectly.
The heuristic was the visible problem. The real one was that our model-based scorer failed often enough for people to see it. Over three days, 125 of 330 optimizations (38%) had stored no score at all.
We deleted the heuristic first. If there's no real score, the UI now says "score unavailable", with no number and no colour. People job hunting right now are anxious enough already. We'd rather show a blank than a number we made up.
Then we had to fix the scorer itself.
Why we stopped asking a chat model for a number
The first version asked a generative model to read the resume and the job description and return a score. That works in a demo. In production we kept hitting three problems:
- The score drifted. The same resume and job description could score 74 and then 81. Users notice when they change one bullet and the score jumps seven points.
- The score and the feedback disagreed. They were generated separately, so the feedback could praise keyword coverage next to a low keyword score.
- The model graded its own homework. The model that rewrote the resume was often the one scoring it.
What I wanted was closer to an exam: fixed questions, typed answers, and the arithmetic done in code.
Two stages, two kinds of model
Stage 1: understand the job. A generative LLM reads only the job description (never the resume) and pulls out a short list of requirements. Each one has a kind, an importance, and a quote copied from the posting.
Then plain code checks that every quote appears in the job description, after normalising whitespace and quote marks. If the model paraphrased or invented a requirement, we drop it. That single check removed a whole category of nonsense, like the "Great Place to Work Certified" badge in a company blurb turning into "candidate must hold a certification".
We cache the result, keyed by a hash of the normalised job description, the extractor version and the model. Without the cache, re-extracting the same posting kept the score within ±3 points only 88% of the time.
Stage 2: answer the questions. For each requirement we ask a decision model a multiple-choice question: does this resume meet it, partly meet it, or not meet it?
The decision model is JEV, a hosted classifier that we call through an early-access decisions API on OpenRouter. It doesn't write text. You give it the evidence and a set of questions, each with named choices and a description of when each choice applies. It returns, for each question, the choice it picked and a probability for every option:
{
"type": "choice",
"choice": "met",
"confidence": 0.91,
"probabilities": { "met": 0.91, "partial": 0.07, "not_met": 0.02 }
}
Code turns those answers into category scores using the probability-weighted mean, applies our rubric, and adds up the total. The model never sees the final number, so it can't get the arithmetic wrong.
The score and the feedback come from the same answers
Here is what I like most about the design. An unmet must-have lowers the score, and the same answer becomes a piece of feedback that quotes the posting: "The job asks for 'experience with Kubernetes in production', and your resume doesn't show it."
Eligibility rules, such as a required graduation window, work as knockouts. If JEV says a rule is clearly not met, the total is capped, however good the rest of the resume is. Under our old generic rubric, failing an eligibility rule cost only a small slice of the score. Applicants would have found that out the hard way, from a rejection email.
What went wrong on the way
A variance check threw away good scores. Our first version discarded the whole score when any category's answers looked too spread out. Education questions on postings with eligibility rules often split between two answers, so the check fired. That one check caused 62 of the 125 missing scores. We removed it, kept the weighted mean, and now log each category's confidence instead.
Generic questions blurred two questions into one. A handful of fixed questions for every posting mixed up "does this job need a degree?" with "does this candidate have one?". JEV answered both badly because we'd asked them as one question. Once the questions came from the posting itself, one per requirement, the answers became decisive. In our testing, choosing the right questions mattered more than any other change.
The confidence field wasn't the probability we needed. We also use JEV to check that a rewritten resume doesn't invent anything. Every line of the draft gets a question: is it supported by the candidate's own material, does it add something they never claimed (a metric, a title, a date), or is the evidence unclear? Our first version compared the confidence field to a fixed cutoff. On a resume copied word for word, confidence came back at 0.31, and the gate blocked real people on their first upload.
The fix was to use the full distribution (simplified here):
// Sum the probability of every "this is invented" choice.
const pUnsupported = unsupportedChoices.reduce(
(sum, c) => sum + (answer.probabilities?.[c] ?? 0),
0,
);
const pSupported = answer.probabilities?.[supportedChoice] ?? 0;
const verdict =
pUnsupported >= rejectMinProbability
? 'unsupported'
: pSupported > 0 && pSupported >= pUnsupported
? 'supported'
: 'abstain'; // never blocks the user
A rejected line gets one repair attempt, and if the repair is rejected too, the line is removed. On our eval set, all 12 faithful resumes came through and all 19 planted fabrications were caught.
A resume tool that invents a certification can cost someone an offer when the interviewer asks about it, so this gate matters more to me than the score. We want every line we write to be something you can defend in an interview.
How we know it still works
Every change to the rubric, the extractor or the JEV model version runs against real resume and job pairs, and has to pass four gates:
- fewer than 5% of pairs unscored
- at least 95% of re-runs within ±3 points
- the mean score moves by no more than ±8
- rank correlation with the previous version of at least 0.7
Each score records which rubric and model produced it. When we bump the version, chats show a small "Scoring updated" note instead of silently changing a number under you.
Cost and speed
I expected this to be expensive. It isn't. One JEV call costs about $0.0002. With 12 to 18 requirement questions it answered in about 450 ms in testing, and the earlier, smaller version ran at a median of about 370 ms in production. We still run it under a hard monthly budget, so a bug can't turn into a surprise bill.
What we keep private
We don't publish the questions, the rubric weights or the thresholds. We keep tuning them, and publishing them would make the score easy to game. The part worth sharing is the architecture. A generative model reads the job, a classifier makes the judgments, and plain code does the maths.
If you've built scoring or grading on top of LLMs, I'd like to hear how you kept it repeatable. And if you're job hunting and a score from any tool, ours included, looks too good to be true, ask where it came from.
來源:DEV Community · dev.to