兩條 Oracle 提示詞規則令 Gemini 2.5 Flash 在 UniQL 首次作答準確率由 35.0% 升至 64.5%

UniQL is a new benchmark that translates BIRD's 1,534 text-to-SQL questions into 16 SQL dialects. Its Oracle column is hard. The best published result is 49.61% (Claude Sonnet 4.5), and Gemini 2.5 Pro scores 26.14%.

I ran schemagate on it with a light model, Gemini 2.5 Flash. The scores below come from UniQL's own evaluator, on Oracle Free 23ai.

UniQL Oracle, 1,534 questions	First try	With error repair
schemagate + Gemini 2.5 Flash	35.0	69.9
+ two Oracle rules in the prompt	64.5	67.6
Best published (Claude Sonnet 4.5, single try)	49.6	—

The first run looked good for the wrong reason. 69.9% was mostly schemagate's repair loop: run the query, show Oracle's error to the model, let it fix it. On a first try alone, Flash got 35%. That's below the bare models, and not a fair comparison with their single-try numbers.

Two mistakes caused almost all of it. Of the 694 failed first queries:

551 wrote FROM account AS a. Oracle doesn't allow AS before a table alias.
100 used a column named DATE or NUMBER without quotes. Both are reserved words.

So schemagate now tells the model Oracle's rules whenever the target is Oracle. First-try accuracy went from 35.0% to 64.5%, about 15 points above the best published model, with a lighter one. Failed first queries dropped from 694 to 85. Other databases' prompts are unchanged.

The lesson I'd take: measure before you tune, and report the number that matches what others report. A repair loop is a fine design, but it doesn't belong in a column of single-try results.

Submitted to the UniQL leaderboard: JerryGao818/UniQL#2. Predictions, evaluator output and the workflow that reproduces it: https://github.com/ashishsinha1602/schemagate/tree/main/benchmarks/results/uniql_oracle