Instructions hidden inside emails made AI assistants generate unauthorized send-email commands in controlled tests. One of the models, Meta SecAlign 70B, had been trained to resist prompt injection. An October 9, 2026 revision of LLMail-Inject reports attack success rates of 58% for GPT-4o, 47% for GPT-5 mini and 32% for Meta SecAlign 70B.
This attack is called indirect prompt injection: an assistant reads a message to summarize an inbox or answer a question, then mistakes instructions in the message for permission from the user. In LLMail, an attack counted only when the model generated an unauthorized email-sending tool call with the required recipient and message body.
Those figures are the authors' controlled evaluation results and have not been independently replicated. The same table reports 22.8% for GPT-5. None of the figures measures compromised users, real mailboxes or the security of a deployed assistant product.
The revised study draws on a collection of 208,095 distinct attack prompts from 839 participants. For the post-challenge tests, researchers replayed collected attacks without adapting them to the target models. But the paper does not disclose the table's exact prompt count, selection rule, number of repetitions, model snapshots or generation settings. The rates therefore should not be read as results from running all 208,095 prompts against each model.
The challenge documentation shows what the attackers had to overcome. The assistant's instructions explicitly prohibited sending mail based on requests found inside received messages. Participants controlled an email's subject and body, then tried to make that untrusted text look like an instruction from the user.
In the original challenge, the hostile email had to reach the assistant, evade the applicable defenses and induce an unauthorized send_email call with the correct recipient and body. Some scenarios supplied the email directly; others required it to appear in search results. One scenario required copying a simulated financial figure from another message. The user and mailbox were simulated, so a successful attempt demonstrated a failure inside the test, not a documented breach.
The released data records retrieval, detection, tool invocation and argument correctness separately. An email could pass a detector yet never produce the required action, so detecting suspicious text and preventing an action are different measurements.
What changed in the October revision
The June 2025 paper already described the challenge, its 839 participants and the 208,095-prompt corpus. October's news is the expanded evaluation, including tests of models beyond those used in the original competition.
- Added: October 2026 results replaying collected attacks against GPT-4o, GPT-5 mini, GPT-5 and Meta SecAlign 70B.
- Changed: The paper expands beyond the June 2025 challenge and defense analysis with a post-challenge benchmarking section.
Why earlier scores looked more reassuring
Meta SecAlign's earlier defense evaluation reported attack success rates of 4.0% on SEP and 1.9% on AgentDojo. Its 32% LLMail result looks far worse, but it does not show that the model became less secure: the tests use different tasks, instructions and success criteria. The earlier evaluations specify generation settings and, for AgentDojo, additional defensive prompting that cannot be matched to the incompletely documented LLMail replay.
The LLMail authors argue that small, fixed collections of template attacks can make defenses appear more robust than they are. Their challenge captured a different process. Participants made 461,640 attempts, received feedback about where each attempt failed and could revise their approach. A second phase strengthened defenses and blocked exact or near-paraphrased earlier attacks, pushing participants to adapt again.
Independent testing supports that general concern. NIST's Center for AI Standards and Innovation reported that targeted red teaming raised attack success from 11% for its strongest baseline to 81% for its strongest new attack. That work used Claude 3.5 Sonnet and AgentDojo-derived scenarios, not LLMail or SecAlign. It shows that attack design can substantially change a security score; it does not establish how much of LLMail's gap came from attack diversity.
The original contest also shows that defenses added substantial friction. The authors report that fewer than 1% of first-phase submissions completed the full attack chain, and that no Phase 2 solve was recorded for GPT-4o-mini sub-levels using the complete defense stack. Those layers had costs: the combined configuration incorrectly flagged 10% of a separate 1,000-email Enron test. The contest defenses are not clearly the same setup used for the newer-model replay, so the results cannot be combined into one security ranking.
For anyone building an assistant with tools, the practical test is whether the complete system can withstand an attacker who keeps changing the wording and trying again. LLMail gives that test more substance, but its replay scores do not answer it for deployed products. The crucial boundary is whether text an assistant reads can gain authority the user never granted.