開源 Text-to-SQL 程式庫 schemagate 在 BIRD Mini-Dev 三種 SQL 方言評測中均排名第一

BIRD Mini-Dev is 500 text-to-SQL questions over 11 databases, in SQLite, PostgreSQL and MySQL versions, scored by whether the query returns the right rows. BIRD keeps the leaderboard and checks the submissions.

schemagate, the Apache-2.0 library I maintain, is now first in all three columns:

	schemagate	Next entry
SQLite	81.0	75.6
PostgreSQL	72.6	69.0
MySQL	72.4	49.2

The setup is plain: schemagate picks the tables a question needs, the prompt carries only those plus BIRD's column notes, and Claude Opus 5.5 writes one query. No candidate voting and no agent loop. If the query errors, the model sees the error and tries again.

schemagate's main job is access control. It removes the tables the caller may not read before the model sees the schema. A benchmark doesn't test that part, but it shows the filtering doesn't cost accuracy.

What it isn't: the main BIRD board, which uses a hidden test set. Our first test result there was 72.11, and a rerun is pending.

Leaderboard: https://bird-bench.github.io/
Run files and commands: https://github.com/ashishsinha1602/schemagate/tree/main/benchmarks/results/bird_minidev
pip install schemagate