PReMind:個人記憶如何結合視覺危險資訊
PReMind: When Personal Memory Meets Visual Hazards
PReMind 提出兩階段強化學習框架,訓練個人化視覺語言助理結合視覺危險證據與用戶記憶,作出合適回應。PReMind-Bench 評估記憶提取、視覺根據、拒答及危險應對,並測試各項能力的聯合表現。4B 版本在相互依存測試集的聯合成功率由基礎模型的 12.9% 提升至 28.6%。
Published on Nov 18
Authors:
,
,
,
Abstract
A diagnostic benchmark and two-phase reinforcement learning framework enable personalized vision-language assistants to jointly connect visual hazard evidence with personal memory for appropriate responses.
Personalized vision-language assistants use visual observations and personal memories to provide advice tailored to the user. When the observed scene contains a physical hazard, personal context, such as allergies, medical conditions, or mobility constraints, can change what help is appropriate. Responding appropriately therefore requires recognizing the visible hazard and applying the relevant person's information. We introduce PReMind-Bench, a diagnostic benchmark that evaluates this combination across diverse hazards and personal-memory conditions. It separately measures memory access, grounding, abstention, and hazard response, and assesses joint success on interdependency cases. Existing models often satisfy one requirement while failing the other, and training only for safety or personalization leaves a gap in joint success. We therefore propose PReMind, a two-phase reinforcement learning framework that combines on-demand and forced-call interactions (memory always retrieved) to jointly train grounded memory use and hazard response. At 4B, PReMind raises joint success from 12.9% to 28.6% over its base on the interdependency set and improves hazard adequacy and correct abstention on the main test. Joint-success gains persist under forced-call evaluation and under a second LLM judge across three Phase II seeds; averaged over these seeds, joint success also improves on unseen hazard groups. These findings highlight the need to evaluate and train personal assistants to connect visual evidence with the relevant person's context.
View arXiv page View PDF Add to collection
Get this paper in your agent:
hf papers read 2611.23808
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 1
Image-Text-to-Text • Updated about 1 hour ago • 8Yeongtak/PReMind-4B
Datasets citing this paper 1
Yeongtak/PReMind-Bench
Viewer • Updated about 1 hour ago • 2.92k • 97
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2611.23808 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co