AgentSafeEvo: Adversarial Co-Evolution for LLM Agent Safety
AgentSafeEvo: Adversarial Co-Evolution for LLM Agent Safety
AgentSafeEvo 提出以攻擊者—防禦者博弈為基礎的協同演化強化學習框架,以提升 LLM 智能體安全性。框架引入由可學習性引導的任務加權,按各任務中防禦者獎勵的變化調整任務對攻擊者和防禦者更新的貢獻。實驗顯示,該方法提升了對抗穩健性,同時維持智能體的一般能力。
Published on Oct 14
Authors:
,
,
,
,
,
Abstract
Large language model (LLM) agents accomplish complex tasks through multi-turn interactions with external tools. However, malicious content embedded in user queries, tool descriptions, or environment observations can manipulate their decisions and induce unsafe actions. Adversarial co-evolution has emerged as an effective approach to strengthening LLM safety, but existing work primarily focuses on language-level safety or indirect prompt injection in web tasks. In this paper, we introduce AgentSafeEvo, a co-evolutionary reinforcement learning framework for agent safety built on an attacker-defender game spanning diverse tasks and attack types. The attacker learns to construct adversarial agent tasks that expose weaknesses in the defender's tool use, while the defender learns to complete tasks safely under these attacks. We further introduce learnability-guided task weighting, which uses within-task defender-reward variability to adjust each task's contribution to attacker and defender updates. Extensive experiments on agent safety and general capability benchmarks demonstrate that our method improves adversarial robustness while maintaining general agent capabilities.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co