Where multi-agent LLM systems fail — 14 failure modes in 1,642 traces
1,642 multi-agent traces, 14 failure modes: the most common was repeating the same step (15.7%); inter-agent misalignment was under a third.
On this page
When a single agent cannot solve a task, a common response these days is to split the work across several agents. Yet in a study that classified 1,642 multi-agent execution traces, the most common failure was not agents misaligning with one another. Repeating the same step (15.7%) and failing to recognize the termination condition (12.4%) ranked first and third, and failures classified as problems between agents made up less than a third of the total.
1. Executive summary
A multi-agent system divides one task among several agents with different roles, such as a planner, an implementer and a reviewer. Splitting the work seems likely to improve results because each agent can focus on a narrower job, but measured results often fall short of that expectation.
7 frameworks · classified by an LLM annotator
problems between agents
vs. chat · Anthropic's own figure
2. Does splitting really help?
The paper by Cemri et al. (arXiv:2503.13657) opens with this sentence: "Despite enthusiasm for Multi-Agent LLM Systems (MAS), their performance gains on popular benchmarks are often minimal." The paper classified failures to find out why.
There is also a counterexample. For its own research system, Anthropic published that "a multi-agent system with Claude Opus 4 as the lead agent and Claude Sonnet 4 subagents outperformed single-agent Claude Opus 4 by 90.2% on our internal research eval." However, this is a figure from the vendor's own internal evaluation, and the same post also states what it costs.
What you gain
For research work that can be split into parallel parts, a 90.2% improvement on the internal evaluation was reported. Subagents explore different branches at the same time.
Source: Anthropic's published post (the vendor's own evaluation).
What you pay
The same post states that "agents typically use about 4× more tokens than chat interactions, and multi-agent systems use about 15× more tokens than chats." Tokens are the basic unit in which a model processes text and in which usage is billed.
Implication: it only works out when the value of a single task exceeds that cost.
3. All 14 failure modes
The taxonomy was built and the failures were counted in separate steps. People built the taxonomy: six experts analyzed 150 traces collected from 5 frameworks, and three rounds of agreement studies reached kappa 0.88 (in each round, 3 annotators independently labeled 5 randomly chosen traces). Kappa measures agreement after removing the chance that raters give the same answer by luck; above 0.8 is generally considered high. By contrast, the 1,642 traces behind the rates below were classified not by people but by an LLM annotator (based on o1). Its agreement with humans was kappa 0.77, and 0.79 on other frameworks that were not used in training.
| Category | ID | Failure mode | Share |
|---|---|---|---|
| System Design Issues FC1 · total 44.2% | FM-1.1 | Disobey Task Specification | 11.8% |
| FM-1.2 | Disobey Role Specification | 1.5% | |
| FM-1.3 | Step Repetition | 15.7% | |
| FM-1.4 | Loss of Conversation History | 2.8% | |
| FM-1.5 | Unaware of Termination Conditions | 12.4% | |
| Inter-Agent Misalignment FC2 · total 32.35% | FM-2.1 | Conversation Reset | 2.2% |
| FM-2.2 | Fail to Ask for Clarification | 6.8% | |
| FM-2.3 | Task Derailment | 7.4% | |
| FM-2.4 | Information Withholding | 0.85% | |
| FM-2.5 | Ignored Other Agent's Input | 1.9% | |
| FM-2.6 | Reasoning-Action Mismatch | 13.2% | |
| Task Verification FC3 · total 23.5% | FM-3.1 | Premature Termination | 6.2% |
| FM-3.2 | No or Incomplete Verification | 8.2% | |
| FM-3.3 | Incorrect Verification | 9.1% |
Source · checked in the original text of arXiv:2503.13657 (all 14 values compared against the PDF text). The per-mode rates are the values as printed in the paper; the category totals are sums of those values (the paper does not state the totals separately). Because of rounding, they add up to 100.05%.
4. The biggest failures were not collaboration failures
When you adopt a multi-agent setup, you might expect poor coordination between agents to be the biggest problem. Yet failures classified as misalignment between agents account for 32.35%. The remaining 67.7% are system design issues or task verification failures.
Looking at the four largest modes makes their nature clearer.
| Failure mode | Share | Can it happen with a single agent too? (our inference) |
|---|---|---|
| Step Repetition | 15.7% | Yes. It appears to come from the agent not being able to track its own progress |
| Reasoning-Action Mismatch | 13.2% | Yes. It appears to come from reasoning and tool calls not matching |
| Unaware of Termination Conditions | 12.4% | Yes. It appears to come from a missing or vague termination condition |
| Disobey Task Specification | 11.8% | Yes. It appears to come from insufficient instructions |
The paper itself writes that "identified failures require more sophisticated solutions," and it saw a gap that is hard to close by refining prompts alone.
5. When to split and when not to
Combining the two sources gives decision criteria. The table below shows criteria we compiled; neither the paper nor the vendor's document presents them as a table.
| What to check | Splitting is reasonable when | Better not to split when |
|---|---|---|
| Task structure | Branches are independent of each other and can run at the same time | Each step's output is the next step's input, so the steps form a chain |
| Context size | The information to handle does not fit in one context window | One context window is enough |
| Tool complexity | Many complex tools have to be handled | Few tools, and simple ones |
| Shared context | Each branch only needs to know its own context | Everyone has to keep the same context throughout |
| Cost headroom | The value of one result justifies using several times as many tokens | Cost-sensitive; 15× the tokens of chat is a burden |
| Current failure modes | The problem is missing things because the search scope is narrow | The problem is repetition, termination conditions or specification violations |
| Example | Research that scans many sources at once | Most coding tasks (named by the vendor itself) |
6. What you can do today
The shares below only tell you where to look first; they do not decide what you should do. They are a distribution of observed failures, not inspection rules. Items 2 through 7 are general responses we chose by looking at that distribution.
| # | Action | How to check |
|---|---|---|
| 1 | Pull the 20 most recent failed runs and classify them into the 14 modes | Which category they cluster in. The table can be used as a checklist as is |
| 2 | Write down the termination condition explicitly | Whether the instructions contain a sentence saying what counts as done. 12.4% of the observed failures were of the termination-condition type |
| 3 | Record progress where the agent can read it | To avoid redoing a step it has already done, the agent must be able to check for itself what it has done |
| 4 | Log what the agent said it would do next to what it actually did | Executing differently from the stated plan was 13.2%. Unless you look at the two side by side, you will not notice it at all |
| 5 | Build a separate verification step | Missing or incomplete verification was 8.2%, and verification that was itself wrong was 9.1% |
| 6 | Make the agent ask for clarification when something is ambiguous | Proceeding without asking for clarification was 6.8%. Write into the instructions when the agent should ask |
| 7 | Compare token use before and after splitting in the same table | Whether your workload shows something similar to the vendor's figure of 15× chat |
7. What this is not yet
- These are not our own measurements. The figures in this report come from arXiv:2503.13657 and Anthropic's published post; we did not measure them again.
- The rates themselves are the output of an LLM annotator. The 1,642 traces were classified not by people but by an o1-based annotator, whose agreement with humans is kappa 0.77. That is lower than the 0.88 among the people who built the taxonomy. As we covered in our report on why both benchmarks and judges are shaky, numbers produced with an LLM as the judge have to be read with that much error in mind.
- The paper is not recent. It was first posted in 2025-03 and last revised in 2025-10. Frameworks and models have changed since then, so it is hard to assume the rates are the same today.
- The category totals are sums we calculated. The paper gives only the per-mode rates and does not state category totals separately. Because of rounding, the three totals add up to 100.05%.
- The 90.2% is the vendor's internal evaluation. It is a figure Anthropic produced on its own research evaluation, and the evaluation set and scoring method have not been published. It is not an independent reproduction.
- "Also happens with a single agent" is our judgment. The last column of the table in section 4 is based on the nature of each mode; it is not the result of the paper measuring them against a single agent.
- The decision criteria in section 5 are our own. The table combines the content of the two sources; there is no such table in either original.
8. Sources
| Source | What we used it for | Link |
|---|---|---|
| Cemri et al., Why Do Multi-Agent LLM Systems Fail? (arXiv:2503.13657) | The 14 modes and their rates (1,642 traces, LLM annotator), the human-built taxonomy (5 frameworks · 150 traces · 6 experts · kappa 0.88), human–LLM agreement 0.77, "performance gains on popular benchmarks are often minimal." Checked in the original abstract, body and methodology | arxiv.org/abs/2503.13657 |
| Anthropic — How we built our multi-agent research system | The 90.2% improvement on the internal evaluation, 4× and 15× tokens, the caveat about domains that need shared context and about coding tasks. The vendor's own figures | anthropic.com/engineering/multi-agent-research-system |
We checked the figures directly in the two sources above. We did not use the widely circulated 42/37/21 category split because it does not match the original. Multi-agent frameworks change quickly, so check the latest documentation of the tools you use before deciding to adopt one.