The End of Determinism: Why Unit Tests Are Useless for AI Agents

In 1936, Alan Turing defined the theoretical machine that would become the foundation of modern civilization. The Turing Machine was absolute. If the machine is in state A and reads symbol 1, it will always write symbol 0 and move to state B. No exceptions. No variance. No negotiation with probability.

Today, we are building systems on a fundamentally different substrate. The Large Language Model is not a Turing machine. It is a probabilistic inference engine — a function that predicts the next token based on a statistical distribution derived from a massive corpus of human knowledge. The shift from Determinism to Probabilism is not a feature toggle. It is a change in the laws of physics for software.

In ordinary engineering terms, this is a test-harness and contract problem. The old unit-test contract was point equality: same input, same output, exact assertion. AI systems need schema checks, evaluation datasets, regression harnesses, statistical acceptance bands, and escalation rules. The hard part is not naming the model probabilistic. The hard part is replacing brittle assertions with a repeatable evaluation contract.

The Stochastic Gap

In Software 1.0 — traditional code — a function is a sealed pipe. Given the same input, it produces the same output. In Software 2.0 — AI systems — the function is a leaky sieve. When we set temperature=0.7, we are asking the machine to roll weighted dice at every token position. We are trading precision for fluency, accuracy for creativity.

The "Sampling" box in Figure 1 is where classical testing assumptions collapse. The same prompt, sent to the same model, at temperature=0.7, will produce meaningfully different outputs across runs. A summarization agent might return "The user is happy" on one run and "The user appears satisfied" on another. Both are correct. Neither matches a hardcoded expected string.

At temperature > 0.0, the LLM is not a function — it is a conditional probability distribution. Every call is a sample from that distribution, not a lookup in a truth table.

Even at temperature=0.0, the problem does not fully disappear. GPU floating-point operations are not always commutative across parallel execution paths. Different batch sizes, different hardware, different CUDA kernel versions — all can shift the output. The determinism you think you have at temperature=0.0 is fragile and infrastructure-dependent.

Why Unit Tests Fail Here

The unit test is built on a single, unexamined assumption: that the function under test is deterministic. Remove that assumption and the entire edifice wobbles. Consider a classification agent that reads support tickets and assigns categories:

def test_classify_billing_ticket():
    ticket = "I was charged twice for my subscription last month."
    result = classify_ticket(ticket)
    assert result.category == "BILLING"

This test passes. You ship. Three weeks later, the model provider silently updates the base model weights. Now the agent returns BILLING_DISPUTE — a reasonable, arguably better answer — and your test suite turns red. Your on-call engineer spends two hours debugging a system that is working correctly.

Unit tests for AI agents don't just fail to catch real bugs — they actively generate false positives that erode trust in the test suite and desensitize engineers to CI failures.

The deeper issue is architectural. A unit test encodes a specific expected output. In a probabilistic system, the space of correct outputs is a region, not a point. Testing a region requires different primitives.

The Three-Layer Evaluation Stack

The Evaluation Harness has three distinct layers, each targeting a different class of correctness.

Layer 1 — Deterministic Assertions. The wrapper code around the LLM is still deterministic. Does the output parse as valid JSON? Does it conform to the Pydantic schema? Is the latency under your SLA threshold? These checks are binary, fast, and should run on every CI commit. Necessary but not sufficient.

Layer 2 — Semantic Similarity. Embed the actual output and a golden reference using the same embedding model. Compute cosine distance. If the distance falls below a task-calibrated threshold, the outputs can be treated as semantically equivalent even when the tokens differ.

Layer 3 — Cognitive Grading (LLM-as-a-Judge). A stronger model grades a weaker model's output against a specific rubric. Effective rubrics are binary: "Did the agent make a factual claim not supported by the retrieved documents? YES or NO." This layer is materially more expensive than the original inference, so it belongs in sampled CI checks and full pre-release reviews rather than every hot-path request.

from openai import OpenAI
import numpy as np
from pydantic import BaseModel
from typing import Literal

client = OpenAI()

# ── Layer 1: Deterministic schema check ──────────────────────────────────────

class AgentResponse(BaseModel):
    category: Literal["BILLING", "TECHNICAL", "RETURNS", "GENERAL", "ESCALATE"]
    confidence: float
    reasoning: str

def layer1_schema_check(raw_output: dict) -> bool:
    """Validates structural correctness. Fast, binary, always runs."""
    try:
        AgentResponse(**raw_output)
        return True
    except Exception:
        return False

# ── Layer 2: Semantic similarity via embeddings ───────────────────────────────

def get_embedding(text: str, model: str = "text-embedding-3-large") -> list[float]:
    response = client.embeddings.create(input=text, model=model)
    return response.data[0].embedding

def cosine_distance(a: list[float], b: list[float]) -> float:
    va, vb = np.array(a), np.array(b)
    return float(1.0 - np.dot(va, vb) / (np.linalg.norm(va) * np.linalg.norm(vb)))

def layer2_semantic_check(
    actual: str,
    golden: str,
    threshold: float = 0.10
) -> tuple[bool, float]:
    """Returns (passed, distance). Distance < threshold means semantically equivalent."""
    dist = cosine_distance(get_embedding(actual), get_embedding(golden))
    return dist < threshold, dist

# ── Layer 3: LLM-as-a-Judge for factuality ────────────────────────────────────

JUDGE_PROMPT = """You are a factuality auditor. Your job is strict and narrow.

SOURCE DOCUMENTS:
{documents}

AGENT ANSWER:
{answer}

Question: Does the agent's answer contain any factual claim NOT supported by the
source documents above? Respond with exactly one word: YES or NO.
If YES, add a newline and quote the specific unsupported sentence."""

def layer3_judge_check(
    answer: str,
    source_documents: str,
    judge_model: str = "gpt-4o"
) -> tuple[bool, str]:
    """Returns (passed, judge_response). Passed = no hallucination detected."""
    prompt = JUDGE_PROMPT.format(documents=source_documents, answer=answer)
    response = client.chat.completions.create(
        model=judge_model,
        messages=[{"role": "user", "content": prompt}],
        temperature=0.0,
        max_tokens=200,
    )
    verdict = response.choices[0].message.content.strip()
    passed = verdict.upper().startswith("NO")
    return passed, verdict

The Golden Dataset

The critical dependency in the evaluation pipeline is the Golden Dataset — a curated collection of (input, ideal output, source documents) triples that represents ground truth. Without it, Layer 2 has nothing to compare against and Layer 3 has no documents to audit. As I argue in The Golden Dataset, if you lose your code but keep your data, you survive. The inverse is not true. A well-maintained annotated dataset is worth more than a large pile of brittle point assertions because it defines the behavior the system must preserve across prompts, models, and providers.

The Architect's Mandate

When a CEO asks, "Why did the AI say that?" — the honest answer is often, "Because it was statistically likely given the prompt and the model's training distribution." You cannot run a bank on "likely." You cannot fly a plane on "probably."

The mandate of the modern AI Architect is not to make the model smarter. It is to build architectures that force reliability out of uncertainty — to bound the failure distribution until it fits inside an acceptable risk envelope.

The engineer who keeps adjusting prompts, hunting for the magic temperature, adding more examples to the system prompt — is fighting the physics. The LLM is probabilistic by design. The entropy is a feature, not a bug; it is what allows the model to generalize across novel inputs rather than memorize a lookup table. The architect's job is to accept the entropy at the model layer and eliminate it at the system layer. The model generates; the system decides. These are different responsibilities requiring different testing strategies.

For how agent state transitions interact with this evaluation model, see The Probabilistic State Machine and The Agentic Contract. The Iron Triangle of AI Systems frames the trade-off precisely: capability, reliability, and cost sit at three corners, and the Evaluation Harness tells you where you actually are on that triangle.

The question that remains open: as agent architectures grow more autonomous — multi-step, tool-using, self-correcting — does the failure distribution converge, or does it compound? The answer determines whether evaluation harnesses scale linearly or whether we need something fundamentally different.

Related: A cross-model benchmark across seven frontier models documented exactly the failure modes determinism would have prevented.

Related Reading

• Temperature 0 Doesn't Mean Deterministic — The mechanism behind nondeterministic outputs
• Seven Models Judged Each Other. The Mistakes Were the Most Useful Part. — Empirical data supporting the end-of-determinism thesis