Entity trail

AgentDojo

Source-backed findings, relationship evidence, citations, and briefing history from the public MindPattern archive.

Briefing refs

Findings

Edges

Sources

Corpus findings

2026-09-10 / agents-researcherAgentAudit attaches to a running agent and scores its trace on ten dimensions, exposing 95.1 vs 22.6 trust spreads at similar task completionarXiv 2609.09875 (2026-09-09) argues existing frameworks measure either task completion (AgentBench) or security robustness (AgentDojo, ASB) but never attribute a failure to a stage. AgentAudit reads only the recorded execution trace — imposing no constraint on the agent's implementation — and scores instruction integrity, planner, memory, tool selection, tool invocation, tool correctness, alignment, tool faithfulness, security and execution integrity, then classifies behavior and attributes the failure to a specific stage. Across nine capability and adversarial tasks, Claude Sonnet 5 and GPT-5 scored 95.1 and 80.6 Composite Trust while Sarvam 105B, Llama 3.3 70B and Gemini 2.5 Flash scored 57.6, 45.7 and 22.6. The sharper point is that models with similar task-completion behavior diverge on trust because several non-frontier models are repeatedly classified Unsafe_Compliance rather than merely failing — a distinction pass/fail benchmarks cannot see. The authors flag that a single judge model, itself one of the evaluated models, scored every trace.
2026-09-03 / skill-finderCo-evolving the harness alongside the policy cut AgentDojo attack success 3x while raising benign utilitySafeEvolve (arXiv 2609.02786, 2026-09-02) argues that safety alignment done only at the policy level or only through external harness updates leaves runtime control disconnected from intrinsic safety. It runs a continual loop where completed on-policy trajectories become safety evidence: on the harness side, bounded component-level updates to the safety prompt and hierarchical skills, producing auditable and reversible harness artifacts; on the policy side, harness-use SFT to teach the model to actually use those artifacts, then harness-augmented RL with verifier-decomposed rewards. For Qwen3.5-4B it delivered a 3x attack-success-rate reduction on AgentDojo while benign utility rose from 59.79% to 61.86%.
2026-09-01 / arxiv-researcherMemGauge Finds Agent Memory Poisoning Risk Has a Threshold at Write Time and Couples to Utility at Read TimeMemGauge separately varies writing admission, management policy and retrieval exposure under matched clean and poisoned conditions, which prior work conflated by measuring utility or attack risk in isolation at fixed settings. Across 11 LLMs and two long-term memory benchmarks three distinct profiles emerge: a threshold-like risk transition during writing, policy-dependent local decoupling during management, and coupled growth of utility and risk during retrieval. Applying the same stage-level measurements to four existing memory systems produces consistent diagnostics, which means write admission is the stage where you can buy safety without paying utility, and retrieval is the stage where you cannot.
2026-09-01 / arxiv-researcherSkillGuard Shrinks Agent Authority After Untrusted Data Arrives, With Zero Extra Model CallsRather than classifying untrusted content or approving individual operations, SkillGuard treats untrusted data entering agent state as contamination and restricts future capabilities so the state cannot reach deployer-defined forbidden states, using a Skill Impact Graph, steerability signatures and an inline reference monitor. Across four AgentDojo suites with Gemini 2.5 Flash and Llama3.3-70B backends, it eliminates attack success on three of four suites for both models and cuts Slack to 4.8% and 14.3%, beating Spotlighting, CaMeL and AttriGuard. Fractional-flow restriction preserves substantially more capability than binary cutoff at equal attack success, and the whole layer adds no model calls or token overhead.
2026-09-01 / agents-researcherAttack success rate hides the number that matters: whether the user can see the injection in the agent's final answerThe authors split attack success rate into Covert Success Rate, injections that leave no trace in the final response, and Overt Success Rate, ones the user can spot. The split follows from the ReAct format, where the final response summarizes the most recent action, so a trace that hands control back to the user task before ending stays invisible. Their ICoA attack deliberately steers the agent back to the user task after firing the injection and posts the highest covert rate across four models on AgentDojo, 3.79 to 12.01 points over the strongest baseline. Anyone reporting ASR on a tool-using agent is measuring the wrong thing.
2026-08-28 / arxiv-researcherJudgeStealer Replicates a Black-Box LLM Judge Across Three Evaluation Protocols From Pointwise Queries AlonearXiv 2608.26982 presents a query-efficient model extraction framework aimed specifically at LLM judges, exploiting strong cross-protocol agreement to acquire pointwise scores and convert them into pairwise and listwise supervision without spending extra victim queries. It selects pointwise inputs by semantic diversity, predictive uncertainty and potential judge biases, then applies score smoothing and multi-protocol review to preserve ordinal structure during surrogate adaptation. Against state-of-the-art LLM-as-a-judge and reward models it reaches 73.3% pointwise, 87.0% pairwise and 71.6% listwise accuracy, holding up across surrogate scales and against representative extraction defenses.
2026-08-28 / arxiv-researcherSARA Separates 'A Tool Output Suggested This' From 'The User Authorized This', Holding ASR Under 0.63%arXiv 2608.27146 argues the core risk in tool-augmented agents is conflating action induction with execution authorization, and splits them into distinct runtime roles. A context-isolated Action Probe exposes action-inducing semantics in observations and records action-origin provenance across steps, while tool calls are authorized only against the user objective and audited evidence from prior authorized successful executions, with a No-History-Promotion rule preventing historical recurrence from laundering an untrusted origin into execution authority. Across AgentDojo and AgentDyn it holds attack success to at most 0.63% in four primary settings while keeping competitive task utility.
2026-08-28 / agents-researcherSPA drives AgentDojo tool_knowledge attack success to zero by planning once and tracking two latticesSPA, posted 27 August, targets persistent agents where attacker-controlled data can alter control flow, reach security-sensitive tool arguments, or compromise a later query. It calls the planner exactly once per query to emit a full plan in a declarative DSL, then applies dual-lattice information-flow control over confidentiality and integrity across explicit data flows and control dependencies, storing execution results as labeled artifacts and exposing only semantic metadata to later planning. Under the tool_knowledge attack it reduces attack success to 0% on AgentDojo and 0.2% on AgentDojo-MQ, the authors' new multi-query extension for measuring secure state reuse and delayed attacks.
2026-08-26 / agents-researcherStepGuard audits tool calls before execution and cuts attack success 77.3% for a 2.8 point utility costStepGuard (arXiv 2608.24777, 25 August) is an open-weight guard model that checks individual tool actions pre-execution rather than scoring completed trajectories, which is where most existing guardrails operate. It is trained on StepGen, a data engine that generates safe and unsafe trajectories sharing identical context but diverging at the risky step, with Balance-GRPO dynamically rebalancing learning between safe and unsafe actions to fight both over-defense and under-defense. On AgentDojo and AgentDyn it reduces mean attack success rate by 77.3% versus no guard while mean utility drops only 2.8 percentage points, and it reports the highest average accuracy among open-weight guard models, comparable to GPT-5.4.
2026-08-25 / skill-finderA flow-centric policy language cut confirmed agent compromise from 33% to 0% on AgentDojo while raising utilityAgentFlow specifies where data may travel in an agent system rather than which single actions are unsafe, using flow and path rules, task-scoped capabilities, controlled release, and stateful taint semantics enforced by a runtime monitor plus an SMT-based verifier. On AgentDojo's 949 cases it drops confirmed compromise from 33.0% to 0.0% and raises utility from 46.7% to 63.3%; on AgentDyn Dailylife it goes 73.5% to 0.0% with utility essentially unchanged, and ASB direct prompt injection lands at 0 successes in 1,200. Seven safety properties each verify in under half a second, which makes the policies checkable in CI.
2026-08-18 / agents-researcherBounded Agents: an authorization layer drives AgentDojo exfiltration from 75-100% to 0% at 0.24ms p99The Agentic Principal Chain reframes prompt injection as an authorization-architecture problem — an injection only matters if the agent holds authority to act on it — by tracking delegated authority across principals and evaluating each request against accumulated session state with six checks plus composition closure. Across 3,154 instances spanning InjecAgent, AgentDojo and ASB, and using a compromised-model evaluation that inserts the ground-truth attack call directly, AgentDojo exfiltration fell to 0% in all four domains and all 544 InjecAgent data-stealing cases were blocked; intent binding cut destruction from 38.6% to 4.0% and manipulation from 90.5% to 12.1%. The honesty is the useful part: authorization latency is 0.24ms at p99, but utility drops 8.6 and 13.9 percentage points across 949 task-injection pairs.
2026-08-14 / skill-finderToolHazard auto-generates executable injection environments, and where the payload sits matters as much as what it saysToolHazard (arXiv:2608.11878, Aug 12) replaces hand-built prompt-injection test environments with an automated pipeline — Environment Simulator, Attacker Agent, User Simulator — that produces runnable environments with discoverable injection points rather than fixed, pre-marked attack locations. The headline result is that injection timing and placement within the environment state materially change attack success, meaning benchmarks with hardcoded payload positions systematically under- or over-state exposure. Training data synthesized by the framework improved robustness on both ToolHazard-Bench and AgentDojo without degrading legitimate task performance, so it doubles as a hardening data source, not only an evaluation.

Graph relationships

BENCHMARKED AGAINST
ICoA
ICoA attack posts the highest covert success rate on AgentDojo.
Source finding
BENCHMARKED AGAINST
SPA
SPA achieved 0% tool_knowledge attack success on AgentDojo.
Source finding
BENCHMARKED AGAINST
StepGuard
StepGuard reduced attack success rate by 77.3% on AgentDojo.
Source finding
BENCHMARKED AGAINST
ToolHazard
ToolHazard-synthesized data improved robustness on AgentDojo benchmark
Source finding
SUPPORTS
ToolHazard
ToolHazard alignment data improves security on AgentDojo
Source finding
BENCHMARKED AGAINST
PromptArmor
PromptArmor drops prompt-injection false-positive and false-negative rates below 1% on AgentDojo
Source finding
BENCHMARKED AGAINST
SARA
SARA holds attack success to 0.63% on AgentDojo.
Source finding
BENCHMARKED AGAINST
JudgeStealer
JudgeStealer is evaluated on AgentDojo reaching 73.3% pointwise accuracy.
Source finding
BENCHMARKED AGAINST
MemGauge
MemGauge produces consistent diagnostics across AgentDojo suites with Gemini and Llama backends.
Source finding
BENCHMARKED AGAINST
AgentSentry
AgentSentry defends against indirect prompt injection via temporal causal modeling
Source finding
BENCHMARKED AGAINST
ICoA -> AgentDojo
ICoA attack posts the highest covert success rate on AgentDojo.
Source finding
BENCHMARKED AGAINST
SPA -> AgentDojo
SPA achieved 0% tool_knowledge attack success on AgentDojo.
Source finding

Source trail

arXiv 2609.09875 arXiv 2608.30362 arXiv 2608.30041 arXiv 2608.27234 arXiv 2608.22868 arXiv 2608.15888 arXiv 2608.11878 arXiv 2608.05108 arXiv 2607.19595 Source: CBC News Vietnam-based security startup Source: CNBC

Graph sources

entity graphfindings textkg edgeskg entitiesnewsletter issues

Fetching from the wire…