Over 5,119 evaluation rows, the model never changed. The same Claude Sonnet backend. The same simulated tool environment. The same 32 attack payloads. The only variable was the orchestration framework wrapping the agent. The result was a 2.6x spread in attack success between frameworks: CrewAI at 11.9%, LangChain at 18.1%, AutoGen at 20.0%, SmolAgents at 31.1%, with a direct API baseline at 15.5%. Tracing the fault lines before the quake hits — this is what a security evaluation looks like when it isolates the layer that actually decides where trust flows. If you are pricing agentic software by model benchmark scores, you are pricing the wrong asset.
The report is called SADF: The Orchestration Framework Is the Attack Surface. It landed at DEF CON 34’s AI Village, and its experimental design deserves more attention than its headline number. The researchers fixed the base model to Claude Sonnet, ran a direct API control group, then measured how four popular orchestration frameworks transform the same model into a more or less penetrable system. Everything ran inside a SimulatedToolEnvironment, so no real credentials or production systems were exposed. The model calls were real; the tools were simulated. That boundary is ethically sound; it is also a reminder that the report is a map, not the terrain.
To understand why this matters, you have to understand what the industry was measuring before. Agent security benchmarks have spent the last two years producing a stream of alignment metrics, refusal rates, and jailbreak resistance scores. Those numbers are useful for model providers, but they treat the model as the only moving part. SADF’s design is an implicit rebuke. By holding the model constant and varying the scaffold, it turns the orchestration framework into the independent variable. This is the difference between testing a car engine on a test bench and testing the car on a road where the driver, the steering, and the brakes all change the outcome. The engine may be fine. The road is not.
The most valuable contribution is not a single failure. It is the taxonomy of eight failure modes: Tool Call Hijacking, Output Poisoning, Cross-Tool Injection, Memory Poisoning, RAG Poisoning, Delegated Authority Abuse, Multi-Agent Propagation, and Context Boundary Violation. Industries do not coordinate after a shock without a shared vocabulary. This is that vocabulary. It is also the beginning of a pricing language: once you can name a failure mode, you can build a control for it; once you have a control, you can put it in a request for proposal.
Then the numbers. A direct API call to Claude Sonnet produced an attack-completion rate of 15.5%. Wrap the same model in CrewAI, and the number drops to 11.9% — below the baseline. CrewAI’s discrete task isolation appears to contain delegation more tightly. LangChain rises to 18.1%. AutoGen reaches 20.0%. SmolAgents jumps to 31.1%, and its breakdown is the real signal: a 20% RAG poisoning rate and a 64% Context Boundary Violation rate. The architecture that maximizes developer freedom also maximizes the paths by which one tool’s output becomes another tool’s trusted input. 64% is not a bug; it is the default behavior of an agent system built without hard context boundaries.
The 20% RAG poisoning rate for SmolAgents deserves its own line. RAG means retrieval-augmented generation; it is how an agent consults external knowledge before acting. If that retrieval channel can be poisoned, the agent is not making a mistake. It is executing the attacker’s instruction while believing it is following the user’s. This is the same logic as a smart contract reading from a manipulated oracle. We learned that lesson in DeFi; now it is being re-learned in agent frameworks.
The second most important correction is methodological. The report found that naive substring matching inflated attack success metrics for Claude models by a factor of four to six. After a refusal-filtered scoring fix, Claude Sonnet’s true attack-completion rate settles at 15.5% and Claude Haiku at 22.3%. Evaluation methodology is not an implementation detail; it is part of the attack surface. A scoring routine that counts a model’s refusal as a successful attack is measuring the model’s patience, not its security.
This is where my own bias needs to be disclosed. I have spent my career auditing failure, from the vesting-schedule bugs that killed 2017-era ICOs to the impermanent-loss models of DeFi Summer. In 2026, I ran a research sprint simulating 10,000 autonomous agents competing for compute in a proof-of-compute mechanism. The failures that kept me awake were never model misalignment. They were agents inheriting another agent’s memory context, tools passing poisoned outputs to each other, and delegation chains that nobody could trace. Reading SADF’s taxonomy was like finding a name for every ghost in that simulation. The model had no idea it was compromised; the orchestration layer had already decided it was.
The commercial read is just as direct. A 2.6x gap between CrewAI and SmolAgents is an arbitrage signal for security budgets. Enterprises building agent pipelines now have quantitative grounds to put framework security into RFPs, procurement reviews, and vendor scorecards. The most likely business model is Security-Evaluation-as-a-Service: an audit product that takes the SADF methodology and turns it into a repeatable deliverable. The CVE evidence already supports the demand side — Azure SRE Agent (CVE-2026-62830) and Langflow (CVE-2026-9198) are real framework-level flaws, not theoretical slides. Companies pay for real vulnerabilities. The only open question is whether the SADF team open-sources its tooling or lets the commercial security giants productize it first. From a macro perspective, security budgets are a form of risk capital. They are priced by the last painful event. This report is an early repricing signal.
The agent framework war will follow the layer-2 playbook. The real difference between OP Stack and ZK Stack was never purely technical; it was the ability to convince a critical mass of projects to deploy first. CrewAI, LangChain, AutoGen, and SmolAgents are not just codebases. They are networks. The framework that can convince the largest number of protocols to deploy on top of it will set the default security posture. The gap between 11.9% and 31.1% will be quoted in procurement rooms, but the real battleground is distribution.
The comparison to the layer-2 wars is not rhetorical. When OP Stack and ZK Stack began their land grab, the technical differences were real but secondary. The first mover that accumulated the most deployed chains set the default interoperability standard. Agent frameworks are doing the same thing, except the cost of the default is not just transaction fees — it is the attack-completion rate of every agent built on top. A framework that wins on developer experience while losing on context isolation will export its attack surface to every protocol that adopted it. That is a hidden systemic risk.
During DeFi Summer, I built Python models to quantify impermanent loss against yield. The lesson was that when a new risk becomes measurable, it becomes tradable. SADF has made orchestration risk measurable. The next step is not a blog post; it is an insurance product or a risk-adjusted yield product that prices frameworks differently. The numbers already exist to price a policy that covers a CrewAI stack at a different premium than a SmolAgents stack.
The macro version of this is easy to miss. Global liquidity is being rotated into AI infrastructure, but the market still treats security as a cost center rather than a credit signal. SADF changes the unit of analysis. Instead of asking “how much compute can this model access?”, a risk committee can ask “what is the attack-completion rate of the framework that controls the compute?” That question belongs in the same credit memo as the collateral ratio, the custody provider, and the settlement layer. The framework is now a counterparty.
The gaps in the report are not reasons to dismiss it; they are reasons to read it carefully. Thirty-two payloads is a small sample relative to the space of possible attacks. If those payloads came from a public benchmark rather than adversarial red-teaming, the attack-completion rates could be optimistic. The claim of eight architectures, with only five detailed data sets, leaves a publishing gap. The earlier version, which claimed coordinated disclosure across ten architectures, has been formally superseded. Configuration sensitivity is unpublished: how much of the spread comes from default framework settings, prompt templates, or tool-permission granularity? And the model generalization question is unsolved. Would the framework ranking hold if the base model were GPT-5.4, DeepSeek, or Llama? The model × framework interaction is the largest unknown on the board. None of this overturns the direction of the finding. This is research-stage evidence, not a standardized evaluation suite, but the method is strong enough to reproduce.
The practical consequence is a new due diligence process. Before any agent is deployed with signing keys, the framework should be tested against the same design: fixed model, direct API control, simulated tool environment, refusal-filtered scoring, and a payload set expanded beyond 32 examples. The absolute numbers will change; the method should not. In production, the attack surface is larger, not smaller. Real credentials, real tool latency, and real APIs add noise and opportunity.
Now the contrarian part. The mainstream framing has been that model safety is the binding constraint, that a safer base model means a safer agent. SADF dismantles that assumption. The same model, the same payloads, the same environment: 11.9% in one framework, 31.1% in another. The model was not the deciding variable. The orchestration layer was. The industry spent eighteen months debating alignment and jailbreaks while the real attack surface was being redesigned every time a developer chose a framework. Collapse is a feature, not a bug — in agent systems, the collapse is simply delivered through the layer that nobody was auditing.
Let me state the decoupling thesis more aggressively. The market is looking for Bitcoin to decouple from the Nasdaq, but the decoupling that matters for this cycle is model security decoupling from system security. The market has acted as if alignment is a sufficient condition for safe deployment. SADF says the opposite: the model can be perfectly aligned and still be weaponized through the orchestration layer. In crypto terms, this is the difference between a secure consensus layer and an unaudited smart contract. The consensus is not the vulnerability; the application is.
The strangest distribution detail: this report surfaced through a blockchain/Web3 news feed. I do not read that as a category error. I read it as an early warning. Crypto protocols are already wiring AI agents into treasury management, trading, and on-chain custody. The next framework-level exploit will drain a smart-contract wallet before a security team finishes reading the incident report. The smart-contract audit industry was born because consensus was secure but application code was not. Agent frameworks are the new application code. Code never lies, but it does omit — and what the simulation omits is real credentials, real tool latency, and real adversarial payloads.
From a liquidity perspective, this is about the collateral behind the AI trade. If AI agents are going to custody assets, move stablecoins, and automate lending, then the framework that governs their tool calls is a form of financial infrastructure. A 64% Context Boundary Violation rate is not an engineering footnote; it is a resolvency risk. In a multi-agent economy, context boundaries are the equivalent of asset segregation in a bank. SmolAgents may be the hot wallet of agent frameworks: convenient, exposed, and catastrophic if funded too deeply.
Even with all those caveats, the technical conclusion is high-confidence. The attack surface is a triad — model, framework, and tool environment — and the framework is the least monitored component. That is the opposite of current security budgets. Liquidity is just patience disguised as capital; it will follow the data. The data says the framework layer was underpriced.
Security budgets are risk capital. They flow to the layer where the last unpriced tail sits. After Terra, that was stablecoin collateral. After SADF, it is the orchestration layer. The next audit report you read should ask not “is this model safe?” but “what framework is carrying the money, and how were the 32 payloads chosen?” Arbitrage is the market’s way of correcting itself. The fault line has been drawn. The only question left is whether your stack sits on the 11.9% side or the 31.1% side.