SHIRITAS

論文 / arXiv:2611.26553

Replication of Agentic Security Events: Audit and Investigation (ROGUE-AI)

PLAIN SUMMARY / やさしい要約

AIが人の代わりに作業するようになると、失敗のニュースも増えます。この研究は、ニュースなどで語られた失敗を「あとから同じ状況を作って確かめる」ための方法を提案します。さらに、原因だと疑われた部分だけを外した別の状況も作り、結果を比べます。たとえるなら、事故の再現実験をして、原因とされた部品だけを交換した車でも同じ事故が起きるかを確かめるような発想です。

AIが専門用語を使わずに書いた解説です。内容の正確さは、下の原文の要旨で確認してください。

ABSTRACT / 要旨(原文)

Agents based on large language models (LLMs) can now delete files and send payments on behalf of their users. News media reports routinely describe failures of these agents (e.g., Replit's coding agent deleting a production database), but a report often only says what went wrong and rarely (in)validates the incident or finds its root cause. Incident catalogs collect these reports, and benchmarks record whether agents fail, but neither replicates an incident nor tests the cause its report might identify. In this paper, we present ROGUE-AI (Replication of Agentic Security Events: Audit and Investigation), a multi-agent framework that closes the above gap and rebuilds each reported incident offline together with a matched control that lacks only the blamed cause, and runs agents on both. Our results suggest that ROGUE-AI is able to reproduce the reported harm in 89% of its 108 attempted replications. Agents built on five current GPT, Claude, Gemini and DeepSeek models made the reported mistake in 18% to 33% of runs, against at most 1% without the blamed cause. Of the 72 incidents with real-world harm, 62 had no underlying attacker, and in these, agents did what their task asked for but missed an ordinary hazard, such as a database shared with the live site. Replications without an attacker caused more harm than replications of prompt injection for all five models. Defenses against prompt injection look for injected instructions, but these replications contain none. Further, a warning after each tool output that names the hazard and its consequence lowered harm for all five models, whereas a note of the same length that does not mention the hazard left most of the harm in place. Built from real incidents, the ROGUE-AI replications form a reusable audit framework that vendors can rerun before each new release.

ここに表示しているのは原論文の要旨です。AIによる要約や解釈は含みません。

この論文を取り上げた記事