The single architecture decision that determines whether a production AI system works is how you compose it. It’s about how many discrete jobs you give to one agent and how many separate agents you use.
Get it wrong in one direction and accuracy collapses. Get it wrong in the other, and the whole system becomes brittle.
I watched this argument get made, and then proven live on stage, by Ashley Nutter, SVP of Product at CNN, at ProductCon New York. CNN is the number one English-language news service in the world, so these lessons come from real production scale.
This piece is written for product leaders and engineers architecting AI systems at scale. By the end, you'll know how to decide the number of AI agents your system actually needs, tier cost by task, and design an architecture that survives the next model release.
Key Takeaways
The single decision that determines whether a multi-agent AI system works is how you compose it: how many discrete jobs you assign per agent.
Over-composing with too many narrow agents fragments context and can hurt accuracy. Under-composing with one agent doing too much makes the system brittle and hard to evolve.
For CNN, single agent given five jobs refused the task or produced broken output, and the same model given one job produced a clean, usable result.
Correctly composed systems let teams tier cost by task, serve different user policies from one shared source of truth, and swap in new models without rebuilding the pipeline.
Anthropic's own internal research found a multi-agent architecture outperformed a single agent by 90.2% on complex tasks, at roughly 15 times the token cost.
Production agentic architecture is a continuous balance of three constraints: accuracy, cost, and evolvability.
What Is Agent Composition in AI Systems?
Agent composition is the decision of how many discrete tasks to assign to a single AI agent versus splitting them across multiple specialized agents. Get the balance wrong in either direction, and the system fails in a predictable way.
Nutter framed the tension better than anyone I have heard. "If you over-compose and you have too many agents, they start to fragment context. That can actually be a hit to the accuracy of the system. If you under-compose and too many things are grouped together, your whole system becomes brittle and difficult to build."
Two quick definitions, because both failure modes have names worth knowing.
Context fragmentation means the work is split across so many AI agents that no single one holds enough shared information to produce a coherent result.
A brittle system is one where the pieces are so entangled that changing any part means risking all of it.
This tradeoff is measurable
Anthropic's engineering team reported that a multi-agent system, with Claude Opus 4 as the lead agent and Claude Sonnet 4 subagents, outperformed a single-agent setup by 90.2% on their internal research evaluation. They noted that multi-agent systems consume roughly 15 times more tokens than a normal chat interaction.
Read those two numbers together, and you get the whole discipline in one sentence. More composition can buy real accuracy, but it is never free. The skill is finding the level of composition your AI use case actually needs.
The Three Constraints Every Production AI Architecture Must Balance
Every production AI decision is a tradeoff across three constraints:
Accuracy, meaning the output is usable;
Cost, meaning it is feasible at the business's real scale; and
Evolvability, meaning it holds up over time without a full rebuild.
Nutter's word for the accuracy bar is the one I want you to steal: "right enough."
The model does not need to be perfect. It needs to be right enough for the specific use case, and that bar sits in a very different place for a social media clip than it does for a broadcast news segment.
In a newsroom, "right enough" is a genuinely high bar rather than a hedge.
A European Broadcasting Union study, cited in the Reuters Institute's report, found that ChatGPT, Perplexity, and Gemini misrepresent news content almost half the time. That is precisely why CNN's architecture includes a dedicated evaluation layer, which we will get to, rather than trusting raw model output.
Cost is the constraint most teams underestimate, because a process that is affordable on a demo becomes ruinous across millions of hours. And evolvability is the constraint most teams ignore entirely, until the day the model landscape shifts underneath them.
Why Video Is the Hardest Input in AI Product Development
Video is one of the messiest inputs in AI product strategy because it is long, unstructured, and actually three data types at once: visuals, sound, and on-screen text and graphics. Until it is deliberately structured, it cannot be searched well, recommended well, or monetized well.
The industry data backs this up. Amagi's June 2026 AIRTIME Report, based on roughly 6,500 streaming channel deliveries and a pulse survey of 28 senior media practitioners, found that 86% of respondents said poor metadata is already costing their organization revenue. The reasons are a weaker discovery and reduced monetization. The same report found 68% expect AI to handle most metadata generation within three years.
Zoom out beyond media, and the problem gets bigger, not smaller. Gartner estimates that unstructured data, including documents and multimedia files, accounts for 70 to 90% of all organizational data. This means most companies are sitting on the same challenge CNN is, just in different formats.
CNN's version of it is extreme by design. Millions of hours of footage from journalists all over the world, growing every day because everyone now carries a camera.
All of it needs to become structured data that scales and stays durable over time. Which brings us to the moment on stage that made the whole argument concrete.
The Artemis Case Study: What Over-Composition Actually Looks Like
Nutter's live example used a real NASA press clip released ahead of the Artemis 2 mission to the moon. She gave one AI agent five related jobs to perform on that clip at once, including finding a usable soundbite, and then showed the audience exactly what happened.
The first attempt failed. As Nutter described it, the model "politely said, 'No thanks. I'm good. I can't do that.'" Five reasonable jobs, bundled into one agent, and the system simply declined.
The second attempt produced a clip with a malformed timestamp. The output did not match the actual footage.
The third attempt found a genuinely moving soundbite about nations coming together and then ruined it. The clip opened with the speaker referring to "this" mission without ever naming it. For an editor who needs a standalone clip, that is an unusable asset.
Then came the fix, and it was almost embarrassingly simple. Nutter gave the same model one job: find a usable clip, instead of five. It returned a clip with the same positive sentiment, except this time the speaker explicitly named the Artemis mission, so the clip made complete sense on its own.
Same settings. The only variable that changed was composition.
This is not a one-off prompting quirk. Chroma's research team tested 18 frontier language models and found that every single one showed measurable performance degradation as input context grew longer. Even when the relevant information was technically sitting right there in the context.
Overloading one agent is a structural problem.
How Right-Sized Composition Pays for Itself
Once the composition is right, three payoffs follow almost automatically. You can tier cost by task, serve different users from one shared foundation, and absorb technology change without a rebuild.
The first two deserve their own walkthrough, and the third deserves its own section. If any of this sounds like the gap on your team right now, Product School's Advanced AI Agents certification covers exactly this kind of agent scoping and orchestration work before you get to the payoffs below.
Advanced AI Agents Certification
Design and implement adaptive, multi-agent AI systems with memory, collaboration, and risk safeguards. Deploy production-ready AI agents at scale.
Enroll now
Composition lets you tier cost by task
Once a task is isolated into its own agent, you can hand it the cheapest input that still works. There’s no need to run everything through the most expensive process available. At millions of hours of scale, you simply can’t afford to make this mistake.
Nutter's example was entity extraction, meaning identifying who appears in a video. Fed only the transcript, which costs a fraction of analyzing the visuals, the model correctly identified three of the four Artemis crew members, because one of them had named the others out loud.
The fourth person, it labeled "unidentified narrator" with the role of voice-over speaker. That person was Reid Wiseman, the commander of the mission. Only the expensive visual pass, which read the NASA patch, the flag, and finally the name on his flight patch, identified him correctly.
The lesson is not that transcripts are bad or visuals are good.
It is that a transcript is right enough for finding soundbites, because a soundbite is literally what somebody said. Entity extraction genuinely earns the expensive pass, and composition is what lets you make that call per task instead of paying a maximum cost for everything.
Composition lets one system serve different users differently
The same use case can require entirely different AI behavior depending on who is asking, and the way to handle that without duplicating your pipeline is to separate a shared baseline source of truth from lightweight, user-specific policy layers built on top of it.
Nutter's example: two users both want to find a transition in the same video.
One wants it for a social media clip and is optimizing for emotion and engagement
The other wants it for ad insertion and is optimizing for brand suitability and risk management.
Same request, opposite definitions of good.
CNN's answer is to build the expensive part once. A baseline chapterization layer finds every transition point in the video, and that becomes the shared source of truth. Then a separate, cheaper policy layer per use case evaluates those same chapter breaks against its own criteria, deciding whether a given break is a great social moment or a safe ad slot.
If those terms are new, the source of truth is the one shared, expensive extraction step you do once. The policy layer is the specific, inexpensive AI evaluation step you run per use case on top of it.
Almost every team I meet is rebuilding extraction for every new use case, which means they are paying the expensive step over and over to answer cheap questions.
If building that evaluation layer is the part your team hasn't figured out yet, Product School's AI Evals certification is built to teach exactly that.
AI Evals Certification
Learn to build trusted AI products. Design eval suites, integrate CI/CD gates, monitor drift and bias, and lead responsible AI adoption.
Enroll now
Why Composition Is Also Your Insurance Against Model Change
A system built from small, well-scoped agents is a system you can partially rebuild when a new model arrives. In this market, that equates to survival.
Nutter described the pace better than any analyst report I have read. "You go to bed one night, and when you wake up in the morning, the entire landscape of capabilities has completely changed."
New models arrive, old ones get deprecated, and capabilities that did not exist last quarter are suddenly table stakes.
The mechanism that protects you is the same composition discipline from the start of this piece. Because each agent has one job and one narrow, specific output, you can swap the model behind any single agent or add an entirely new agent. You don’t even need to touch the rest of the pipeline.
Modularity is how the system absorbs a technology shift that nobody can schedule.
The full picture at CNN is a stack of three layers.
Metadata layer of composed agents outputs the structured data
Product layer builds editor-facing experiences on top of it
Evaluation layer, including human review, feeds quality signals back into the system so it improves over time and catches the new errors.
If you want to go deeper on that last layer, we have covered how AI Builders and AI PMs should run AI evals in detail.
Nutter was honest about the bill for all this, and I want to be too. Orchestrating many agents introduces real complexity, particularly around schema management and tracking how one agent's error propagates into the agents downstream. AI-native product teams are intentional about both.
Her closing thought is the mental model I keep repeating to product leaders. "AI is not a feature so much as a layer that we can actually build products on top of."
Features get shipped and forgotten. Layers get built on for a decade.
How to Apply This to Your Product
You do not need CNN's archive to apply this. You need four moves, made in this order, on whatever system you are building next.
Write down the discrete, specific outcome each agent is responsible for, before building anything. If you cannot state an agent's job in one sentence, it is not one agent's job, it is several. This is the agent-scoping discipline Product School's Advanced AI Agents certification teaches, for teams ready to move past single-agent AI prototypes into orchestrated systems.
Separate your source of truth from your policy layers. Do the expensive, shared extraction once, then build a cheap, specific evaluation layer per use case or user segment on top of it. If you find yourself rebuilding extraction for every new use case, this is the move you skipped.
Tier your inputs by task. Default to the cheapest input that reliably produces a right enough result, and reserve the expensive pass for the tasks that genuinely need it, the way a transcript serves soundbites but not entity extraction. Building the evaluation muscle to know the difference is precisely what our AI Evals certification covers.
Design for swappability. Ask what breaks if you replace the model behind one agent, and if the answer is five other systems, that is a sign of under-composition.
Here is the checklist I would run a team through before signing off on a production AI architecture:
Move | Ask your team | You are ready when |
Define discrete outcomes | Can we state this agent's one job in a single sentence? | Every agent in the system has exactly one clearly defined output |
Separate source of truth from policy | Are we rebuilding extraction for every new use case? | One shared baseline layer feeds multiple lightweight, use-case-specific policy layers |
Tier cost by task | Are we defaulting to the most expensive process for everything? | Each task runs on the cheapest input that reliably produces a right enough result |
Design for swappability | What breaks if we replace the model behind one agent? | Swapping a model touches one agent's scope, not the whole pipeline |
AI Is a Layer, Not a Feature
The decision that determines whether production AI works is not which model you choose, because the models will keep changing whether you like it or not.
It is how you compose the system around them:
one job per agent
one shared source of truth beneath many policy layers
an architecture where any single piece can be swapped without touching the rest.
That is what "AI is a layer" means in product reality. It is the mental model I would take into your next planning cycle. If you are ready to build this kind of system yourself, Product School's Advanced AI Agents certification is the structured place to learn it, live and hands-on.
Advanced AI Agents Certification
Design and implement adaptive, multi-agent AI systems with memory, collaboration, and risk safeguards. Deploy production-ready AI agents at scale.
Enroll nowUpdated: October 16, 2026
