JFinBench:可持續更新的日本金融基準測試,題目取自最新資格考試
JFinBench: A Continuously Refreshable Japanese Financial Benchmark from the Latest Certification Exams
JFinBench 是一個可定期更新的日本金融基準測試,涵蓋會計、税務、理財規劃及銀行業,收錄 2026 年六項定期舉行的資格考試中的 321 道題目。測試 12 個近期大語言模型時,Claude Fable 5 和 GPT-6 Astra 的整體分數均超過 93%,最佳開放權重模型 Kimi K3.1 Pro 則低 Claude Fable 5 12 個百分點。
Published on Nov 13
Authors:
,
Abstract
Large language models (LLMs) are increasingly applied to finance, creating a need for benchmarks that evaluate their capabilities across financial systems with different regulations and practices. However, existing financial benchmarks provide limited coverage of country-specific financial systems, and they are difficult to keep up to date as released questions leak into training data. We introduce JFinBench, a Japanese financial benchmark comprising 321 examples from the 2026 editions of six regularly administered certification exams covering accounting, tax, financial planning, and banking. To construct it, we develop a human-in-the-loop agentic pipeline in which a coding agent writes exam-specific conversion code and a review page, while a human verifies every example and feeds corrections back to the agent. This pipeline adapts to heterogeneous exam formats and reduces the effort of incorporating new exam editions, enabling periodic refreshes that mitigate data contamination. Evaluating 12 recent LLMs, we find that Claude Fable 5 and GPT-6 Astra exceed 93% overall, while even the best open-weight model, Kimi K3.1 Pro, trails Claude Fable 5 by 12 percentage points. When models can additionally use a terminal and web access, they retrieve the knowledge they lack and every model improves. The gap narrows but about 4 points remain, and the ranking of models stays largely the same, suggesting that knowledge held in the model itself remains valuable even with tools. We release the dataset, the evaluation code, and the construction pipeline.
View arXiv page View PDF Project page GitHub 0 Add to collection
Get this paper in your agent:
hf papers read 2611.18130
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2611.18130 in a model README.md to link it from this page.
Datasets citing this paper 1
SakanaAI/JFinBench
Viewer • Updated 30 minutes ago • 264
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.18130 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co