Multi-Agent Systems That Hold Up Under Real Workloads
Multi-agent systems look magical in demos and collapse the moment they meet real-world data. The difference between a prototype and a production system is almost never the model — it is the architecture around the model.
If your agent system has no shared state object, no deterministic control flow, no critique step, and no model routing — it is a demo, not a system.
The Problem
Agent frameworks make it trivially easy to wire up multi-step LLM workflows that look impressive in a notebook. Two weeks into production they exhibit the same pathologies:
- Non-reproducible behavior between runs of the same input
- Infinite loops or runaway tool calls
- Costs that scale with the bug count, not the user count
- Failures that are impossible to root-cause because nothing is replayable
Why Typical Implementations Fail
The root cause is almost always that control flow has been delegated to the LLM. When an agent decides which agent to call next, you have lost the ability to reason about your own system. The model is non-deterministic. Your control flow is now non-deterministic. There is no debugger for that.
The Four Pillars
1. Explicit State
Treat agents as pure functions over a shared state object. Every transition writes to state, every read goes through state. The state object is typed, serializable, and persisted. Persist it and you can replay any failed run. That single property turns post-mortems from guesswork into a debugger session.
2. Deterministic Orchestration
The graph of "which agent runs when" is code, not an LLM decision. LLMs choose tools; they don't choose control flow. Mix those up and you lose the ability to reason about behavior, set retry budgets, or guarantee termination.
3. Validation Loops
Every agent output passes through a critique step that can accept, reject, or escalate. The critic is cheaper than the producer — that is the whole trick. When possible, replace the LLM critic with a deterministic validator (schema, regex, business rules). LLM critics are a fallback, not a default.
4. Cost-Aware Model Routing
Frontier models for hard reasoning, small models for everything else. Default to the cheapest model that passes eval; escalate only when the validator rejects twice. Done correctly this is a 5–10x cost reduction with no quality loss.
Tradeoffs
- Explicit state is verbose. It is also the foundation everything else stands on.
- Deterministic orchestration removes "emergent" behavior. That is the goal.
- Routing requires eval. Without eval you cannot tell which tier each task belongs in.
Expected Outcomes
Replayable runs. Bounded cost and latency. Self-correcting outputs. Root-cause analysis that takes minutes instead of days. And — the property clients actually pay for — behavior that does not silently change between Tuesday and Friday.
Key Takeaways
- State is the contract; everything else is plumbing
- Control flow belongs in code; reasoning belongs in the model
- The critic is the reliability layer; build it before you scale
- Model choice is a routing decision, not a foundation