ultraielts 每道 IELTS 題目發布前均經盲解檢查

Disclosure: this post was written with an AI assistant, working from the project's code and docs.

I run ultraielts, a free site where people preparing for IELTS take full mock tests in the computer-delivered format. The interface is in Turkish because that's who it's for. The tests are in English, as on the real exam.

The worst bug a practice test can have isn't a crash. It's a question with two defensible answers, or an answer key that's simply wrong. A student marks NOT GIVEN, the key says FALSE, and from then on they stop trusting every score the site gives them. The tests on the site are original (written for it, no third-party material), and a pipeline helps write them, so I needed a check that is stricter than me rereading the key at 2 a.m.

So every listening part and reading passage goes through a blind solver before it can be published.

The solver sees only what a candidate sees

For each reading passage or listening part, a model gets exactly what a candidate gets:

• the passage, or for listening, the transcript as "what the candidate hears"
• the questions without answers, produced by the same function that strips the keys before the site sends a test to the browser
• any figures (maps, diagrams, charts) as images

It's told to answer as a band 9 candidate would, using only the text, obeying each group's word limit, and to give a confidence level, a reason of at most 25 words that quotes the decisive words, and a note when something about the question itself makes it impossible or ambiguous.

Reusing the public "strip the answers" function matters more than it sounds. If the solver got a slightly different view than students do, it would be testing a different exam.

Scored by the site's own scoring code

The solver's answers aren't compared to the key with string equality. They're scored by the same scoreGroup function that scores real students. Word limits like "NO MORE THAN TWO WORDS AND/OR A NUMBER", spelling variants, optional words in brackets and TRUE/FALSE/NOT GIVEN aliases are all judged exactly as they would be for a person.

That removes a whole class of false alarms. "colour" against a key of "color", "7:30" against "7.30", or "forty-five" against "45" isn't a mismatch, because the scorer accepts all of those from a student. When the scorer wouldn't accept the solver's answer, I want to know before a student finds out.

Two runs, three kinds of disagreement

The solver runs twice, at two different reasoning-effort settings. Comparing the runs with the key gives three useful signals:

• Strong mismatch: both runs disagree with the key.
• Weak mismatch: only one run does.
• Consensus: both runs give the same wrong answer. This is the interesting one. It usually means the key is missing an acceptable variant, or the key is wrong.

A reviewer decides what the mismatch means

A mismatch is a symptom, not a verdict. A second pass, the reviewer, sees the whole unit with the keys, the deterministic validator's findings, and every mismatch with both runs' answers and reasons. It reports issues with a severity and a suggested fix, and gives each mismatch a verdict: the key is wrong, a variant should be added, the question is ambiguous, the key is right after all, or it's unsure.

The verdict sets the severity:

It also looks for things the solver can't: answers that break the word limit, NOT GIVEN vs FALSE confusion, listening answers that come out of order in the recording, and distractors too weak to distract anyone.

Repairs are allowed, within strict rules

Errors, and warnings the reviewer marks as auto-fixable, go to a repair step, one question group at a time. The repair has to return the corrected group plus precise edits: exact find-and-replace edits for passages, line edits for transcripts, and a changelog. A repair is rejected if:

• it changes any question numbers,
• its edits don't apply cleanly to the current text, or
• the result has more validator errors than before.

After a round, only the groups that changed are solved and reviewed again. Anything the checkers no longer report is marked fixed. There are at most three rounds. Anything still open stays open, and the test waits for a human.

Each issue's id is a fingerprint of where it was found and what it is (source, test, section, issue code, group, questions). The same finding keeps the same id across rounds and across later QA runs, so once I've looked at something and waived it, it stays waived instead of coming back every time.

The publish gate

Publishing is a separate command that refuses when:

1. the test would be public but isn't original or rights-cleared (this one can't be waived),
2. the validator still finds errors, or
3. the test's QA job has open error-level issues, unless I waive them by code.

The rights rule is the one I care about most. Test-prep sites are full of "real past papers" of doubtful origin. Here, anything imported from a book stays private, as a share link for one's own students, and only original or rights-cleared tests can be public.

What I took away

• Don't let the checker see more than the user does. A solver that sees the key, or a different rendering of the question, is checking something else.
• Score with production code. If your checker has its own notion of a correct answer, you end up with two notions of correct.
• Separate symptoms from verdicts. The solver only finds disagreements; a different pass with more context decides what each one means.
• Constrain the fixer harder than the finder. Most of the safety comes from what a repair is not allowed to do.

The site is live, free and needs no signup: ultraielts. The interface is Turkish, but the tests are English. If you write exam questions of any kind, I'd like to hear how you check them.