All articles
Whitepaper

Pokee AI Benchmarks Enterprise Agent Security with DTap

Pokee AI uses DTap to measure prompt-injection resistance, direct misuse, and useful task completion in a complete agent system.

By DecodingTrust Team

DTap puts attack resistance and useful task completion on the same evaluation footing. Pokee AI uses it to assess Pokee-Isaac 28B v0.1 as a complete agent system across 14 domains, covering direct misuse and indirect prompt injection.

That is an important distinction for enterprise adoption. An agent operates with a particular model, scaffold, set of connectors, and permissions. Teams need to understand how those pieces behave together when a user asks for legitimate work and when an attacker tries to redirect it. DTap provides the shared environments and outcome checks needed to study that behavior as a system.

A common testbed for comparing agent systems

The white paper includes a controlled comparison across 12 Linux domains, with the same judge. Pokee-Isaac records 8.3% indirect attack success, 30.5% direct attack success, and 85.6% benign task success. It runs in its own scaffold, while the five comparison models use the stock runner; these are system-level measurements.

The original six-system comparison table from Pokee’s white paper, showing direct ASR, indirect ASR, combined ASR, benign task success, and serving routes.
Table 8. Macro-average over the same 12 Linux domains and judge, sorted by combined ASR (lower is safer). Pokee-Isaac uses its own scaffold.

Reporting benign task success alongside attack success makes the comparison useful to a deployment team. A system that refuses more work can look safer without serving its users better. The paper’s DTap evaluation keeps both measurements visible, so readers can examine whether a security result comes with a change in practical capability.

Following the complete agent stack

The paper’s reference architecture shows why an agent needs end-to-end testing. A decision can travel through planning, memory, tools, retrieval, and enterprise systems before its effects become visible. Testing a prompt alone misses much of that path.

Pokee’s original enterprise agent architecture, showing client surfaces, the agent harness, tools, retrieval, serving, and systems of record.
Figure 1. Today’s enterprise agent stack, drawn without security modules. Solid arrows are the request path; dashed arrows are telemetry.

Turning evaluation into concrete security work

The hardened architecture adds eight security modules around those boundaries. For teams using DTap, the follow-up questions are practical: which boundary did an attack cross, what action occurred, and does a mitigation preserve the user’s legitimate task? The complete trajectory and resulting state give developers something concrete to investigate.

Pokee’s original hardened architecture, with eight security modules added to the same enterprise agent stack.
Figure 2. The same architecture with security modules baked in. Green blocks are added; grey blocks are unchanged from Figure 1.

A practical role for DTap in enterprise evaluation

Pokee AI’s study illustrates how DTap can support an enterprise security assessment: define the permitted task, introduce a direct request or an indirect injection, and examine what the agent actually does. The resulting traces and outcome checks make the assessment concrete enough to inform engineering work, rather than ending at a leaderboard position.

The same evaluation pattern is relevant when teams compare models, adjust a tool interface, or add a new security control. Retaining the tested configuration and reading attack success together with benign utility helps them explain a change in behavior. The white paper is one published example of DTap being used to connect agent security research with the decisions involved in building enterprise systems.