作者以 SIGMOD'27 公開基準評測 schemagate 的 Text-to-SQL 權限控制
I ran my text-to-SQL access control on a benchmark I didn't write
作者以 SIGMOD'27 論文公開的基準評測 schemagate 的 Text-to-SQL 角色權限控制，使用 Claude Sonnet 4.5 執行五次測試，並採用論文作者的評分器。
Every number I had published for schemagate came from my own tests. This month a SIGMOD'27 paper released a public benchmark for exactly the problem schemagate exists for: Benchmarking Text-to-SQL under Role-Based Access Control. So I ran it.
The test
Each question comes with a role and a list of the columns that role may read. If the answer needs a column the role can't see, the right reply is "Sorry, I cannot answer." The authors score it with their own evaluator. I didn't change a line of it.
Their main finding is uncomfortable for tools like mine. If you just hide the restricted columns, models still answer. They patch something together from the columns that are left. On BIRD, their best "hide the schema" setup still answered 27% of the questions it should have refused.
What schemagate does differently
Three things happen for every question:
- The role's policy becomes visibility. The model never sees a hidden table or column, not even its name.
- One line tells the model that its view is partial, without saying what is missing. If the question needs data it can't see, it should refuse instead of improvising.
- Every query passes a scope check before it runs. SQL that touches anything outside the policy is refused.
Results
Same model as the paper's best runs (Claude Sonnet 4.5), five trials, the authors' scoring:
| AC-F1 | answered when it should refuse | SQL that read forbidden data | |
|---|---|---|---|
| BIRD, schemagate | 90.1 | 5.4% | 0 |
| BIRD, paper's best | 88.1 | 7.4% | not reported |
| Spider, schemagate | 93.0 | 5.1% | 0 |
| Spider, paper's best | 93.0 | 4.6% | not reported |
On BIRD it beats the best configuration in the paper. On Spider it ties. In neither did any query read data the role wasn't allowed to see. The scope check makes that a guarantee rather than a measured rate.
And the paper's best setup shows the model the whole schema plus the policy. schemagate gets these numbers without the model ever seeing what is restricted.
What each piece is worth
Taking the scope check out doubled the bad answers on BIRD (5.4% to 11.3%). The one-line notice was tested on 100 questions: 16% bad answers without it, 5% with it.
One surprise: giving the model BIRD's column descriptions made things worse. Better notes on the visible columns helped it find workarounds for questions it should have refused.
The honest caveats
- I chose the notice's wording on 100 BIRD questions that are part of the test set. Spider played no part in any choice, so it's the cleaner of the two results.
- One model, where the paper tested thirteen.
- My BIRD databases are the 2024 release; the paper used the 2025 one.
The notice ships on by default in schemagate 1.6.0. To reproduce the runs: benchmarks/rbac_text2sql.py, plus the evaluator's reports in benchmarks/results/rbac/.
pip install schemagate · GitHub
來源：DEV Community · dev.to