跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 18 天前AI 評分60

大型語言模型不知道自己知道甚麼:FinDendro 基準評估金融回測中的時間自我知識

LLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting

AI 導讀

研究團隊提出 FinDendro 基準,以 2,226 條涵蓋 1999 至 2026 年的有日期美國事實,評估大型語言模型的時間知識及其自我判斷。GPT-5 對受控事實的答對率為 76%,但只聲稱知道其中 38%;它對自己表示不知道的事實,仍有 61% 答對。研究發現,知識聲稱可超出測得邊界最多七個季度,因此,訓練窗口內的回測不能單憑模型自述、提示詞或供應商標示的知識截止日期確認沒有資訊洩漏。

正文

Published on Nov 4

Authors:

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Financial backtests of large language models (LLMs) are meaningful only if the model did not already know the outcomes it is asked to predict. Practitioners guard against this by trusting a vendor's stated knowledge cutoff, by asking the model what it knows, or by instructing it to answer as of a past date. To verify whether any of these safeguards can be trusted, we introduce FinDendro, a benchmark of 2,226 dated U.S. facts from 1999 to 2026, covering monthly macroeconomic series, month-end stock prices and Federal Reserve decisions. Each fact yields four prompts: a value question, Recall and Predict questions that pair each candidate value with a claim of knowing or guessing without a declared date, and a Predict question with a historical declared date. Controls for persistence and dominant outcomes keep chance accuracy near 0.27. Results show that for most frontier models, self-reports understate knowledge. GPT-5 answers 76% of controlled facts correctly but claims to know 38%, and answers 61% of the facts it says it does not know. Knowledge claims can persist beyond the measured knowledge boundary, by up to seven quarters, and a declared date removes most of the claims of prior knowledge while leaving accuracy unchanged. Backtests inside a model's training window therefore cannot be certified leak-free by self-report, instruction or stated cutoff. These findings motivate behavioral measurement of temporal knowledge boundaries, and FinDendro provides a benchmark and procedure for selecting backtest periods based on demonstrated factual knowledge.

View arXiv page View PDF Add to collection

Get this paper in your agent:

hf papers read 2611.06808

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.06808 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.06808 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.06808 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co