The Iron Triangle of Inference: You Can't Have Smart, Fast, and Cheap
In distributed systems, the CAP Theorem dictates you can only have two of Consistency, Availability, or Partition Tolerance. In AI, we face a new, equally brutal constraint: The Iron Triangle of Inference. You must choose between Quality, Latency, and Cost. You cannot have them all.
In distributed systems engineering, we have the CAP Theorem. It states that during a network partition, you can preserve only two of three properties: Consistency, Availability, or Partition Tolerance. Brewer didn't publish it as advice — he published it as a constraint of physics. You don't argue with it. You design around it.
Agentic AI systems have an equivalent constraint. Every architectural decision — which model to call, how many prompts to chain, how much context to retrieve — collapses into a trade-off between three vertices: Quality, Latency, and Cost. The engineers who ignore this triangle ship systems that are either brilliant and bankrupt, fast and shallow, or accurate and agonizingly slow.
This is the Iron Triangle of Inference. And like the CAP Theorem, it is not a suggestion.
- Every inference call trades off Quality (reasoning depth), Latency (TTFT + generation time), and Cost (token price × volume). You can optimize for any two — never all three simultaneously.
- Three production patterns emerge: the Genius Pattern (Quality + Latency), the Triage Pattern (Cost + Latency), and the Deep Thought Pattern (Quality + Cost).
- Routing routine traffic to cheaper tiers and reserving frontier calls for complex tasks can materially reduce blended cost versus a single-model architecture.
- Intelligence Arbitrage — dynamically routing tasks to the cheapest model capable of solving them — is the core architectural skill in production agentic systems.
The Triangle Is a Law, Not a Leaderboard
The Iron Triangle reframes model selection entirely. Instead of "which model is best?", the correct question is: which vertex of the triangle does this specific task inhabit?
- Quality (Intelligence): Reasoning depth, instruction adherence, hallucination rate. The strongest available reasoning models live here, but they are rarely the right default for routine work.
- Latency (Speed): Time to first token and total generation time. Synchronous user-facing applications have much less tolerance for slow reasoning than batch jobs, back-office agents, and offline review systems.
- Cost (Economy): Dollars per token multiplied by request volume. At high request volume, even a modest per-call difference becomes an architecture constraint rather than a procurement detail.
The Three Patterns in Production
Pattern 1: The Genius Pattern (Quality + Latency)
You need a strong answer while the user is still waiting — a contract review during a live customer call, a clinical triage recommendation, a fraud review on a material transaction. Call the strongest appropriate model with a well-structured prompt and minimal intermediate chaining. Complexity in the prompt chain adds latency without automatically adding intelligence.
The cost is real. This pattern is viable when the transaction value, risk reduction, or user experience justifies premium inference. It breaks when a product quietly uses frontier reasoning for routine traffic because nobody separated exceptional cases from normal ones.
Pattern 2: The Triage Pattern (Cost + Latency)
Classification pipelines, intent detection, spam filtering, routing decisions, embedding generation — these tasks do not require frontier reasoning. Use distilled, small, or high-throughput models where the task is bounded and the evaluation harness proves quality is stable.
Triage models can be excellent classifiers and poor reasoners. The relevant boundary is not model prestige; it is whether the task remains bounded enough for cheap evaluation to catch mistakes.Pattern 3: The Deep Thought Pattern (Quality + Cost)
You need high accuracy, but synchronous latency is not the binding constraint. Scientific hypothesis evaluation, complex code generation with correctness checks, multi-document synthesis — these are Deep Thought tasks. The implementation uses deliberate loops: draft, critique, retrieval, ensemble voting, or verifier passes over cheaper model calls.
The cost is latency and operational complexity. This pattern belongs in batch processing, background agent tasks, research pipelines, and review queues. It does not belong in every chat turn just because the architecture can do it.
The Architect's Tool: Intelligence Arbitrage
Intelligence Arbitrage means dynamically routing tasks to the cheapest, fastest model capable of solving them. I've explored the economics in detail in Intelligence Arbitrage: The Economics of Routing Thoughts. The architecture has two components: a lightweight classifier that determines task complexity, and a router that maps complexity to the appropriate model tier.
A production-representative implementation using LangGraph and LiteLLM:
from enum import Enum
from typing import TypedDict
from litellm import completion
from langgraph.graph import StateGraph, END
class ComplexityTier(str, Enum):
TRIAGE = "triage" # simple, low-stakes, classifiable
FRONTIER = "frontier" # complex reasoning, high-stakes
DEEP_THOUGHT = "deep_thought" # accuracy-critical, latency-tolerant
class AgentState(TypedDict):
user_input: str
complexity: ComplexityTier | None
response: str | None
MODEL_REGISTRY = {
ComplexityTier.TRIAGE: "provider/fast-classifier-model",
ComplexityTier.FRONTIER: "provider/frontier-reasoning-model",
ComplexityTier.DEEP_THOUGHT: "provider/mid-tier-reasoning-model", # used in loop
}
CLASSIFIER_PROMPT = """Classify the complexity of this user request.
Return exactly one word: TRIAGE, FRONTIER, or DEEP_THOUGHT.
TRIAGE: greetings, simple lookups, yes/no questions, routing decisions
FRONTIER: contract review, legal analysis, complex reasoning, multi-step plans
DEEP_THOUGHT: scientific analysis, code correctness proofs, multi-doc synthesis
User request: {input}
Classification:"""
def classify_intent(state: AgentState) -> AgentState:
"""Fast, cheap classification using a small model."""
response = completion(
model=MODEL_REGISTRY[ComplexityTier.TRIAGE],
messages=[{
"role": "user",
"content": CLASSIFIER_PROMPT.format(input=state["user_input"])
}],
max_tokens=5,
temperature=0.0,
)
raw = response.choices[0].message.content.strip().upper()
tier_map = {
"TRIAGE": ComplexityTier.TRIAGE,
"FRONTIER": ComplexityTier.FRONTIER,
"DEEP_THOUGHT": ComplexityTier.DEEP_THOUGHT,
}
state["complexity"] = tier_map.get(raw, ComplexityTier.FRONTIER)
return state
def route_by_complexity(state: AgentState) -> str:
return state["complexity"].value
def handle_triage(state: AgentState) -> AgentState:
response = completion(
model=MODEL_REGISTRY[ComplexityTier.TRIAGE],
messages=[{"role": "user", "content": state["user_input"]}],
max_tokens=256,
)
state["response"] = response.choices[0].message.content
return state
def handle_frontier(state: AgentState) -> AgentState:
response = completion(
model=MODEL_REGISTRY[ComplexityTier.FRONTIER],
messages=[{"role": "user", "content": state["user_input"]}],
max_tokens=2048,
)
state["response"] = response.choices[0].message.content
return state
def handle_deep_thought(state: AgentState) -> AgentState:
"""Three-pass CoT loop: draft > critique > final answer."""
model = MODEL_REGISTRY[ComplexityTier.DEEP_THOUGHT]
# Step 1: Draft with explicit reasoning
draft = completion(
model=model,
messages=[{"role": "user", "content": f"Think step by step. {state['user_input']}"}],
max_tokens=2048,
).choices[0].message.content
# Step 2: Self-critique
critique = completion(
model=model,
messages=[
{"role": "user", "content": state["user_input"]},
{"role": "assistant", "content": draft},
{"role": "user", "content": "Identify any errors or gaps in your reasoning above."},
],
max_tokens=1024,
).choices[0].message.content
# Step 3: Final corrected answer
final = completion(
model=model,
messages=[
{"role": "user", "content": state["user_input"]},
{"role": "assistant", "content": draft},
{"role": "user", "content": critique},
{"role": "user", "content": "Produce a final, corrected answer."},
],
max_tokens=2048,
).choices[0].message.content
state["response"] = final
return state
# Build the graph
graph = StateGraph(AgentState)
graph.add_node("classify", classify_intent)
graph.add_node("triage", handle_triage)
graph.add_node("frontier", handle_frontier)
graph.add_node("deep_thought", handle_deep_thought)
graph.set_entry_point("classify")
graph.add_conditional_edges("classify", route_by_complexity, {
"triage": "triage",
"frontier": "frontier",
"deep_thought": "deep_thought",
})
graph.add_edge("triage", END)
graph.add_edge("frontier", END)
graph.add_edge("deep_thought", END)
app = graph.compile()
The classifier is intentionally narrow: it needs one routing label, not creative prose. The Deep Thought handler makes sequential calls to a mid-tier model rather than one frontier call. Whether that is cheaper depends on provider pricing and token shape, which is exactly why the route map has to be recalibrated instead of hard-coded once.
A classifier that returns TRIAGE with high confidence is different from one that barely avoids FRONTIER. The confidence threshold is an illustrative policy, not a universal constant. For high-stakes work, default-route upward when confidence is weak; the cost of an unnecessary frontier call is usually lower than the cost of a wrong triage answer on a complex task.
The Cost Cliff
The financial argument for Intelligence Arbitrage is multiplicative, not linear. In many agent systems, request complexity is lopsided: routine classification and formatting dominate volume, while frontier reasoning represents a smaller but high-stakes slice. The exact mix has to come from logs, not vibes.
The planning model is simple:
C_blended = p_triage*C_triage + p_frontier*C_frontier + p_deep*C_deep + C_router
If most requests are routine and the price gap between tiers is large, routing changes the unit economics. The mechanism is not "use worse models." It is "stop paying frontier prices for work the frontier was never asked to do."
Intelligence Arbitrage is not a cost-cutting tactic — it is a structural acknowledgment that different cognitive tasks have different computational requirements, and that treating all tasks identically is as architecturally unsound as using a sledgehammer for every fastener.The triangle also intersects with determinism. When you route a task to a Deep Thought loop, you increase output variance — each additional reasoning step is another opportunity for the model to diverge. This is the subject of The End of Determinism, where I argue the correct response is not tighter control but better constraint design. And understanding why linear chains fail — why probabilistic routing is the correct mental model — is the next layer, which I develop in The Probabilistic State Machine: Why Linear Chains Fail.
The question every architect should be asking is not which model to standardize on. It is whether their routing policy will still be correct six months from now — because the triangle's edges shift every time a new model drops, and the system that cannot recalibrate is the system that bleeds money in silence.
Related Reading
- The End of Determinism — The triangle constrains a non-deterministic design space
- Intelligence Arbitrage — Arbitrage operates within the triangle's constraints
- The Probabilistic State Machine — State transitions face the same cost-quality-speed tradeoff