All articles
Evaluation

Meta Uses DTap to Red-Team Muse Spark 1.3

DTap provides realistic environments and adaptive attacks for Meta’s tool-use and browser-use safety evaluations.

By DecodingTrust Team

Meta uses DecodingTrust-Agent in its Muse Spark 1.3 safety evaluation to test whether agents can be redirected while using tools and browsers. DTap brings the target environments and adaptive red-teaming into the same evaluation workflow.

For an agent team, a release decision needs evidence about the actions a system can take. A convincing answer in a chat window is only part of that picture. Once an assistant can read an inbox, edit a document, or operate a browser, testing needs to follow the interaction through to its effects in the environment.

One platform, two threat models

DTap-Red searches for both harmful requests and instructions hidden in external content. Meta adapts DTap’s tool interfaces to the API and Muse harnesses, evaluating both configurations against adaptive attacks.

Testing direct misuse with DTap

On the DTap direct-misuse evaluation, Meta reports 5.6% attack success for the API configuration and 10.0% for Muse.

Meta’s original DTap direct-misuse chart, including model comparisons and error bars.
Figure 25. DTap Direct Misuse Benchmark. We report the Attack Success Rate (ASR).

Testing prompt injection with DTap

DTap also tests attacks delivered through tools and environments during legitimate tasks. Both Muse configurations report 4.0% tool-use ASR; browser-use ASR is 17.5% for API and 10.0% for Muse.

Meta’s original DTap prompt-injection chart with browser-use and tool-use results, legend, and error bars.
Figure 29. DTap Prompt Injection Benchmark. Attack success (pass@1) across the tool-use and browser-use subsets.

From model testing to industry workflows

DTap combines realistic task environments, an autonomous attacker, and verifiable outcomes. Its simulated services cover familiar workflows in products such as Google Workspace, PayPal, and Slack. This gives engineering teams a place to test an agent’s interactions with external information and tools, while keeping the task and the security goal explicit.

Meta’s assessment and Pokee AI’s enterprise security study show two complementary industry applications: evaluating a frontier agent configuration and comparing an enterprise agent system. Across both, the practical question is what happens when an agent is given tools, access, and an attacker-controlled input. DTap supplies a common setting for investigating that question.

Evidence teams can act on

For a team reviewing a new model, tool integration, or agent scaffold, the useful output is a connection between the attempted attack, the resulting tool actions, and the task outcome. That context helps developers decide which boundary to inspect and what to test again after a mitigation. It also keeps a security score connected to the system that produced it.

For evaluators, the workflow is reusable: put an agent in a realistic task, adapt an attack, and inspect the resulting actions. DTap makes that workflow available across models and agent configurations, with outcomes that can be examined and retested.