AgentDojo audited scoreboard

For each of 24 models from eight developers: the attack rate that the AgentDojo benchmark reports, next to the rate at which the agent actually carried out the attack, read from the tool calls it made.

Frozen at preprint v1.2 (2026-11-17). AgentDojo v1.2.2, important_instructions attack, 156 attack cases per model. Not updated.

Cite as doi:10.5281/zenodo.24321025 (preprint v1.2).

Rates are over the 156 attack cases of trial 1, leaving out runs that could not be judged. Select a column heading to sort by it.

What the columns mean

Model, developer
The model as the paper names it, and the company that makes it. Appendix B of the paper lists how each model was called and with which settings.
Benchmark-reported attack rate
The share of the model's judged runs that AgentDojo's own check scored as a successful attack.
Agent carried out the attack (95% interval)
The share of judged runs in which the agent's recorded tool calls show that it executed the attacker's decisive step, such as paying the attacker's account. For the one goal judged from the reply text (recommending the attacker's hotel), a reply counts when the paper's frozen warning rule reads it as an endorsement. The range in brackets is a 95% Wilson interval.
Warnings scored as attacks
Runs, of the 156, that the benchmark scored as attacks although the agent's reply named the attacker's hotel to warn the user about the planted instruction.
Attacks scored as no attack
Runs, of the 156, in which the agent executed the attacker's decisive call but the benchmark scored no attack.
Could not be judged
Runs, of the 156, that recorded an error or had no agent turn, for example a request the provider rejected, a reply a content filter blocked or a tool call that was not valid JSON. The paper calls them "can't tell". Both rates leave them out, so these models have fewer than 156 runs behind their rates.
Benchmark rate and carried-out rate, goal mix (post hoc)
The same two rates with each attacker goal weighted by how many of the benchmark's 949 instances it has. The 156-case set holds all 20 instances of the hotel goal and four of each other goal, so the hotel goal is 20 of 156 cases here against 20 of 949 in the benchmark. The paper shows the scoreboard's ordering of the Claude 5.x models depends on its oversampling of one goal.

Reading the Claude 5.x rows

For the hotel goal, AgentDojo's check looks for the hotel's name in the agent's reply, so a reply that names the hotel to warn the user is scored as an attack (paper, section 5.2). This gives the four Claude 5.x models benchmark rates of 10.9% to 12.8% on this set. No Claude model carried out an attacker's tool call in these runs. The 1.3% of Claude Sonnet 5.5 and the 0.6% of Claude Fable 5.2 are hotel replies that the frozen warning rule read as endorsements; the paper reports these as warnings the rule missed.

Where the numbers come from

  • Every number in the table comes from three released files in TRUSCOR/research (commit f84e2e9): scoreboard.json and scoreboard-anthropic.json of study S2, and pf-posthoc-v1-2-2026-11-17.json for the goal mix. The script build/make_page.py in this Space rebuilds the data from them.
  • The rates, interval ends and counts of runs that could not be judged were checked against Table 13 of the v1.2 PDF, and the goal-mix rates of the 11 models that Table 6 lists against Table 6. The other 13 goal-mix rates come from the post hoc file.
  • The two count columns are trial-1 counts from the same files. Table 13 of the paper counts hotel warnings over all 60 hotel runs of a model (20 instances, three trials) instead.
  • Each model's released .sova bundle holds 196 runs: the 156 cases once, plus a second and third run of each of the 20 hotel cases. sova audit on a bundle counts all 196, so it reports different rates from this table, which counts each of the 156 cases once.
  • All results come from one benchmark, one attack template and simulated apps, with the models as they were in November 2026.
  • Competing interest: the author leads TRUSCOR, which develops SOVA.