Deterministic Testing The LLMs

The first version of an LLM application usually has a test that looks suspiciously like this:

response = openai.chat.completions.create(...)
assert "refund" in response.choices[0].message.content

It works on Tuesday.

On Wednesday, the model replies with “reimbursement” instead of “refund”.

The test fails, nobody changed the code, and the build pipeline develops the temperament of a household cat: technically functional, but governed by forces beyond human understanding.

This is the point where many teams discover an uncomfortable truth

Calling a real language model is not the same thing as testing an LLM-powered system.

A real model is valuable for evaluation, exploratory testing and final integration checks. It is usually a poor default dependency for unit and component tests.

That is why I built mock-llm, a deterministic, offline Python library for testing LLM-powered applications without network calls, API keys or external services. It supports text and chat completions, streaming, tool calls, structured output, embeddings, failures, multi-call scenarios and an OpenAI-compatible HTTP transport.

This article explains why that category of tool is necessary

An LLM pipeline is still a software system

Consider a moderately realistic AI workflow:

User request
   ↓
Prompt construction
   ↓
LLM classification
   ↓
Tool selection
   ↓
External API call
   ↓
LLM synthesis
   ↓
Structured response

Only two stages may involve a language model directly.

Everything around them is conventional software:

Yet teams often test the entire pipeline by calling a real model and hoping it says approximately the right thing.

That is equivalent to testing a payment service by calling a real bank cli every time a developer runs pytest.

It may feel realistic. It is also slow, expensive and strangely committed to chaos.

The problems with real-model tests

1. The output is nondeterministic

Even with a temperature of zero, model providers may change infrastructure, model revisions, decoding behaviour or safety filters.

A test expecting:

{"category": "billing"}

may receive:

{"category": "payments"}

Both might be semantically reasonable. Only one satisfies the assertion.

This creates a dangerous habit: developers weaken assertions until almost any output passes.

Eventually the test becomes:

assert response is not None

Congratulations. The model has returned something. Civilisation may continue.

The purpose of a unit test is not to prove that an LLM provider is alive. It is to prove that your code responds correctly to a known input.

2. Failure cases are difficult to reproduce

Production LLM calls fail in interesting ways:

• rate limiting

• malformed JSON

• content-filter rejections

• context-window overflows

• invalid tool arguments

A proper mock lets a test declare:

First call: rate limited
Second call: rate limited
Third call: success

The repository’s scenario API supports multi-call behaviour such as retry-then-success, permanent failure, malformed JSON, slow responses, partial streams, context-limit errors and tool-call sequences.

That gives the test control over the story.

Without it, the provider writes the plot, and the provider has rejected your screenplay.

3. Tests become slow and expensive

A healthy unit-test suite should run frequently like before a commit, during code review , in CI etc.

If every test calls a live local or hosted model, the suite becomes slow, costly, rate-limited, dependent on provider uptime, and burdened with secret management.

Once tests become ceremonially expensive, they are performed about as often as ceremonial sword polishing.

Fast tests get run. Slow tests get described in onboarding documents.

4. Real-model tests blur responsibility

Suppose a test fails because the final classification changed.

Where is the defect?

Prompt construction?
Model behaviour?
JSON parsing?
Provider SDK?
Network transport?
Retry implementation?
Business mapping?

When all layers are tested together, failures provide little diagnostic value.

A good test suite separates responsibilities.

Unit tests
    Test application logic with deterministic mock responses
 
Component tests
    Test HTTP and SDK integration with a mock transport
 
Contract tests
    Verify compatibility with the provider’s expected schema
 
Evaluation tests
    Measure quality against real models and labelled datasets
 
Smoke tests
    Make a small number of real provider calls

Different test types answer different questions

Now what should we mock?

The answer is not “everything”.

Mock the LLM boundary when testing deterministic application behaviour.

For example, suppose an application classifies support tickets:

def route_ticket(text: str, llm) -> str:
    result = llm.invoke(f"Classify: {text}")
 
    if result.content == "billing":
        return "payments-team"
 
    return "general-support"

The unit test should control the model output:

from mock_llm import MockLLM
 
def test_billing_ticket_is_routed_to_payments():
    llm = MockLLM()
    llm.when_contains("refund").respond("billing")
 
    team = route_ticket("I need a refund", llm)
 
    assert team == "payments-team"

This test is not checking whether a real model understands refunds.

It is checking whether the application correctly maps billing to payments-team.

That is an application responsibility, so the test should be deterministic.

Mock behaviour, not implementation trivia

A common mocking mistake is coupling the test to every character of a prompt.

For example:

llm.when_exact(
    "You are a helpful classifier.\n"
    "Classify the following request...\n"
)

This may break whenever someone improves spacing or wording.

Prefer matching the behaviourally meaningful parts:

llm.when_contains("refund").respond("billing")

Or use a predicate:

llm.when_predicate(
    lambda messages: any(
        "refund" in message["content"].lower()
        for message in messages
    )
).respond("billing")

mock-llm supports exact, case-insensitive, substring, regular-expression, full chat-sequence, system-prompt, predicate and invocation-number matching.

The matcher should be as strict as the contract requiresbut no stricter.

A test that breaks because someone added “please” to a prompt is not protecting the system. It is guarding punctuation with the seriousness of airport security.

Strict mode catches accidental gaps

Mocks can become dangerously permissive when they silently return empty or default responses.

Suppose a prompt changes from:

Classify this refund

to:

Determine the billing intent

A loose mock may no longer match, but the test might continue using a default response (set as part of mock initialisation).

Strict mode changes this behaviour:

llm = MockLLM(strict=True)
llm.when_contains("refund").respond("billing")

If no rule matches, the library raises a diagnostic NoMatchError. The rule-resolution chain otherwise checks explicit rules, ordered responses, a configured default, strict failure and finally an empty fallback.

Strict mode is especially valuable in unit tests because unexpected prompts should usually fail loudly.

Silent fallback is polite in demos and suspicious in tests.

Mocking does not replace evaluation

This distinction is essential.

A mock proves that your system behaves correctly when the model returns a known result.

It does not prove that a real model will produce that result.

You still need evaluation for questions such as

Does the classifier choose the correct category?
Does the summariser preserve important facts?
Does the agent select the right tool?
Does the model resist prompt injection?
Does a new prompt improve quality?

Those belong in a labelled evaluation suite, typically run against selected real models (modern evals).

Think of it this way:

Mock tests:
If the model does X, does our system respond correctly?
 
Evaluations:
How often does the model actually do X?

Both are necessary.

Mocking without evaluation can produce a perfectly tested system built around an imaginary model.

Evaluation without mocking produces an intelligent demo balanced on a tower of flaky tests.

The testing pyramid for LLM systems

A practical distribution looks like this:

                ┌─────────────────┐
                │ Real-model E2E  │
                │ Few, deliberate │
                └────────┬────────┘
                   Evaluations
              Model quality and safety
             ┌───────────┴───────────┐
             │ Component/contract    │
             │ Mock HTTP transport   │
             └───────────┬───────────┘
                  Unit tests
       Deterministic MockLLM responses
    ┌────────────────────┴────────────────────┐
    │ Prompt handling, retries, tools, state, │
    │ validation, routing and error recovery  │
    └─────────────────────────────────────────┘

Unit tests should remain fast and deterministic.

A similar layer should test transport and SDK compatibility.

A deliberately limited set should call real models (modern evals)

Do not invert this pyramid unless your CI budget has recently inherited oil fields.

A senior-engineering rule of thumb

Before writing an LLM test, ask:

Which uncertainty am I trying to test?

If the uncertainty is in your application logic, mock the model.

If the uncertainty is in the provider contract, test the transport boundary.

If the uncertainty is in model quality, run an evaluation.

If the uncertainty is whether the production service is reachable, run a smoke test.

Do not use the most expensive and nondeterministic test type merely because it feels “more real”.

A production-grade LLM pipeline is not one that calls a real model everywhere.

It is one that knows exactly where reality and causality is required