跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 19 天前AI 評分56

Kierston 與 KierstonBench:模擬賓夕法尼亞州阿倫敦,並評估語言模型在刑事調查與監控任務中的表現

Kierston & KierstonBench: Simulation of Allentown, PA and Evaluation of Language Models in Crime Investigation and Surveillance Tasks

AI 導讀

Kierston 與 KierstonBench 建立城市規模模擬環境及評測基準,16 個語言模型中只有少數前沿模型能完成大部分調查任務。Kierston 模擬約 125,000 名阿倫敦居民的日常活動,並生成電訊、金融、行政及車牌紀錄,模型須透過受限紀錄介面跨來源連結線索。多數模型提出紀錄不支持的身份判定;少數表現較好的模型仍會受常見欺騙影響,沒有模型取代調查人員。

正文

Published on Nov 3

·

Submitted by

C

on Nov 4

Authors:

Abstract

Kierston and KierstonBench provide a simulated city-scale environment to evaluate large language models on criminal investigation tasks using linked surveillance records, revealing that only frontier models perform reliably while remaining vulnerable to deception.

Large language models are candidates for collecting and analyzing records in criminal investigations and mass surveillance. Their capability and accuracy in this work cannot currently be measured. Real deployments lack the ground truth needed to score a model's conclusions, and no benchmark evaluates these capabilities in a simulated environment. We present Kierston, a city-scale simulation engine, and KierstonBench, a benchmark that tests these capabilities against known simulated events. Kierston models approximately 125,000 residents of Allentown, Pennsylvania, generating their daily activities and the telecommunications, financial, administrative and license-plate records that surveillance and investigative systems collect about them. These records derive from a common behavioral timeline and are distributed across sources representing separate providers. Models investigate through a restricted records interface, linking observations across sources without access to the underlying simulation. The benchmark comprises 32 task families covering record linkage, routine reconstruction, incident investigation and assessment of deceptive accounts. In an evaluation of sixteen language models, most could not perform records investigation and submitted identifications the records did not support. A small number of frontier models solved most of the tasks and abstained when evidence was insufficient. In some cases these models still fell for common deceptions and assumptions, and none replaced an investigator.

View arXiv page View PDF GitHub Add to collection

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.04785 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.04785 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.04785 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co