Testing LLM Workflows Locally

(Repo and examples can be found here)

Knowing that an LLM should be mocked is the easy part.

The harder question is how to mock it without creating a cardboard model of reality that says “success” regardless of what the application does and how it evolves.

A useful LLM mock must reproduce the behaviours your software depends on. For example :

Chat messages
Structured responses
Tool calls
Streaming events
Retries
Token limits
Embeddings
Failures across multiple invocations

Otherwise, the test passes locally and production immediately discovers an exciting new interpretation of JSON.

The mock-llm repository approaches this through several layers

Input matchers
      ↓
Rules and responses
      ↓
MockLLM facade
      ↓
Scenarios and failure simulation
      ↓
Optional OpenAI-compatible HTTP transport

Internally, the project uses a facade around an invocation engine, strategy-based matchers, fluent rule and scenario builders, chain-of-responsibility rule resolution, and an adapter-style HTTP transport.

Let us use those capabilities to build a practical test strategy

Start at the narrowest boundary

Assume we have a ticket-classification service:

class TicketClassifier:
    def __init__(self, llm):
        self.llm = llm
 
    def classify(self, text: str) -> str:
        response = self.llm.invoke(
            f"Classify this support ticket: {text}"
        )
        return response.content

The narrowest useful test injects MockLLM directly:

from mock_llm import MockLLM
 
def test_refund_is_classified_as_billing():
    llm = MockLLM(strict=True)
    llm.when_contains("refund").respond("billing")
 
    classifier = TicketClassifier(llm)
 
    assert classifier.classify(
        "My refund has not arrived"
    ) == "billing"

This is the ideal unit-test boundary.

There is no HTTP server, SDK or network transport. The test verifies only the classifier’s interaction with its LLM dependency.

Notice the use of strict=True. If the application unexpectedly changes the prompt so that the configured rule no longer matches, the test fails rather than quietly receiving an empty response.

Tip: A strict mock is like a senior reviewer, mildly inconvenient and usually preventing something worse.

Choose matching rules deliberately

Different tests require different matching precision.

Exact matching

Use exact matching when the full prompt is part of the contract

llm.when_exact("Return READY").respond("READY")

This is appropriate for stable protocol messages but brittle for evolving prompts.

Substring or regex matching

Use semantic anchors for most application tests

llm.when_contains("refund").respond("billing")
llm.when_regex(r"order-\d+").respond("order_support")

Chat-sequence matching

Use full message matching when roles and conversation order matter

llm.when_chat([
    {"role": "system", "content": "Classify tickets."},
    {"role": "user", "content": "Refund missing"},
]).respond("billing")

Predicate matching

Use a predicate when the requirement cannot be expressed cleanly as text matching

llm.when_predicate(
    lambda messages: (
        messages[0]["role"] == "system"
        and "JSON" in messages[0]["content"]
    )
).respond('{"category": "billing"}')

The library provides exact, case-insensitive, substring, regex, chat-sequence, system-prompt, predicate and call-number matchers. Its rules are resolved in order, with the first matching rule winning.

That last detail matters.

Place specific rules before broad ones:

llm.when_contains("urgent refund").respond("priority_billing")
llm.when_contains("refund").respond("billing")

Reversing them makes the broad rule swallow the specialised case.

Computers follow precedence rules with impressive discipline and absolutely has no interest in your intent.

Test structured output as a contract

Most mature LLM applications should avoid parsing arbitrary prose whenever the output is intended for software.

Suppose the application expects:

from pydantic import BaseModel
 
class Classification(BaseModel):
    category: str
    confidence: float

A successful test can return the model directly:

llm.when_contains("refund").respond_model(
    Classification(
        category="billing",
        confidence=0.97,
    )
)
 
result = llm.invoke_structured(
    "Classify this refund",
    response_model=Classification,
)
 
assert result.category == "billing"

The repository supports structured responses from Pydantic models, dictionaries and JSON, along with deliberate invalid responses through respond_invalid() and invalid model responses.

The invalid case is at least as important:

def test_invalid_classification_is_rejected():
    llm = MockLLM()
    llm.when_contains("refund").respond_invalid(
        '{"category": 14}'
    )
 
    with pytest.raises(StructuredOutputError):
        llm.invoke_structured(
            "Classify this refund",
            response_model=Classification,
        )

This verifies your handling of schema violations.

Do not test only the model’s good manners. Production has a broader emotional range.

Model retries as a sequence, not a loop with hope

Retry logic is difficult to test with independent static responses.

A scenario makes the expected sequence explicit:

from mock_llm import (
    MockLLM,
    MockLLMRateLimitError,
    Scenario,
)
 
scenario = (
    Scenario("retry-classification")
    .on_call(1)
    .raises(MockLLMRateLimitError())
    .on_call(2)
    .raises(MockLLMRateLimitError())
    .on_call(3)
    .responds("billing")
)
 
llm = MockLLM(scenario=scenario)

Now the application can be tested against a controlled timeline:

Call 1 → rate limit
Call 2 → rate limit
Call 3 → success

The scenario system represents calls as numbered steps with actions such as responding, raising, streaming or returning tool calls.

The library also includes presets for common behaviours including immediate success, retries, permanent failure, malformed JSON, slow responses, partial streams and context-limit failures.

This allows separate tests for

1. success after retry;

2. retry exhaustion;

3. non-retryable authentication failure;

4. timeout backoff;

5. context-length fallback.

A robust retry test should assert more than the final response:

assert llm.call_count == 3

It should also verify that authentication errors are not retried and that failure metadata is preserved.

Retries are a reliability mechanism, not a slot machine.

Test tool calls as typed commands

Agentic applications frequently ask a model to choose a tool.

For example:

llm.when_contains("weather").respond_with_tool_call(
    name="get_weather",
    arguments={"city": "Sydney"},
)

The application can then dispatch the requested function and assert the interaction:

response = llm.invoke(
    "What is the weather in Sydney?"
)
 
assert response.tool_calls[0].function["name"] == "get_weather"
assert llm.last_call.called_tool("get_weather")

The repository supports single and multiple tool calls, custom IDs, tool-call streaming, multi-turn loops, argument assertions and invalid-argument testing.

A good tool-calling test suite should cover:

Correct tool, correct arguments
Correct tool, missing argument
Unknown tool
Multiple tools
Tool execution failure
Tool result returned to the model
Maximum tool-loop limit

The last case is easily forgotten.

Without a loop limit, two components can politely ask each other for more information until the data centre reaches retirement age.

Test streaming as a protocol

Streaming is not simply “the same response but gradually”.

It introduces additional behaviour

Chunk ordering
Empty chunks
Partial output
Finish signals
Cancellation
Formatting over HTTP

A simple stream test could be

llm.when_exact("story").respond("hello world")
 
chunks = list(llm.invoke_stream("story"))
 
assert "".join(
    chunk.content for chunk in chunks
) == "hello world"

But also test failure after partial content:

Chunk 1 → "hello"
Chunk 2 → " world"
Chunk 3 → timeout

The library supports word-based and fixed-size chunking, custom streams, delays, async streaming and simulated mid-stream failures.

Your application must decide whether partial output should be:

• displayed

• discarded

• marked incomplete

• retried

• persisted for diagnostics

There is no universal answer, but there should definitely be an answer.

Test SDK integrations through the transport layer

Direct injection is ideal when your application owns the LLM abstraction*

Sometimes it does not.

A service may use the OpenAI Python SDK, LangChain LiteLLM, raw httpx or another library that expects OpenAI-compatible HTTP endpoints.

Monkey-patching internal SDK methods is fragile because it couples tests to library implementation details.

MockLLMTransport instead plugs into httpx and handles OpenAI-style chat completions, text completions, embeddings and SSE streaming entirely in process. It requires no mock server or network port.

For example:

from openai import OpenAI
import httpx
 
from mock_llm import MockLLM, MockLLMTransport
 
llm = MockLLM()
llm.when_contains("refund").respond("billing")
 
transport = MockLLMTransport(llm)
 
with httpx.Client(
    transport=transport,
    base_url="http://test",
) as client:
    openai = OpenAI(
        http_client=client,
        api_key="sk-test",
    )
 
    response = openai.chat.completions.create(
        model="mock-llm",
        messages=[
            {
                "role": "user",
                "content": "Classify this refund",
            }
        ],
    )
 
assert response.choices[0].message.content == "billing"

This is a component test rather than a pure unit test.

It verifies that

1. your SDK configuration is correct;

2. the expected endpoint is called;

3. the payload shape is valid;

4. the response is parsed correctly.

No uvicorn, no port conflicts and no ritual sacrifice to localhost.

Make embeddings deterministic

RAG pipelines introduce another nondeterministic dependency: embeddings

Calling a real embedding model in every retrieval test is unnecessary when the behaviour under test is just

Vector storage
Similarity ordering
Metadata filtering
Top-k handling
Duplicate suppression

The repository includes deterministic embeddings with configurable dimensions and two modes: hash-based vectors and token-feature vectors. The latter is designed to make shared tokens influence similarity, while repeated input produces identical output.

from mock_llm import DeterministicEmbedding
 
embeddings = DeterministicEmbedding(
    dimensions=384,
    seed=42,
)
 
first = embeddings.embed("refund request")
second = embeddings.embed("refund request")
 
assert first == second

For retrieval tests, use controlled documents and assert ordering.

Do not assert that the fake embedding has the same semantic quality as a production embedding model. It will not.

The purpose is to make the retrieval machinery deterministic, not to fit a neural network into a trench coat.

Inspect history, not just output

A test should often verify how the LLM was used:

assert llm.call_count == 1
assert "refund" in llm.last_call.input_text

History and assertion helpers can expose:

• prompt content;

• system prompts;

• tool arguments;

• invocation count;

• response metadata;

• usage information.

The repository records each invocation and provides helpers for checking calls, tools and system prompts. It also supports simulated token usage and context-window enforcement.

This is particularly useful for preventing accidental regressions such as

Calling the LLM twice
Omitting a required system instruction
Sending sensitive data
Exceeding the intended context
Invoking the wrong tool

Use interaction assertions selectively. Testing every comma in a prompt makes refactoring painful.

Test the contract, not the author’s preferred line wrapping.

Use pytest fixtures to standardise test setup

The package includes pytest fixtures such as:

mock_llm
mock_async_llm
mock_embeddings
mock_llm_factory
mock_llm_scenario

These reduce repeated setup and give teams a shared testing vocabulary.

A project-level fixture can establish stricter defaults:

@pytest.fixture
def strict_llm():
    return MockLLM(strict=True)

Another fixture can provide standard scenarios for rate limits, authentication failures and malformed output.

This prevents each team from inventing its own slightly different simulation of the same provider failure.

Consistency is a feature. Especially when the alternative is seven bespoke interpretations of HTTP 429.

A practical test portfolio

For one LLM-backed feature, aim for a portfolio like this:

Unit tests

Use direct MockLLM injection for:

• routing;

• parsing;

• state transitions;

• fallback behaviour;

• tool dispatch;

• retries.

Component tests

Use MockLLMTransport for:

• OpenAI SDK integration;

• request payloads;

• streaming over SSE;

• embedding endpoints.

Evaluation tests

Use real models for:

• classification quality;

• groundedness;

• extraction accuracy;

• prompt comparisons.

Smoke tests

Use a small number of real calls for:

• credentials;

• deployment configuration;

• provider reachability.

The key is not to eliminate real models from testing.

It is to stop asking real models to test things that deterministic software can test better.

Closing thought

A reliable LLM test suite should be able to say:

We know what happens when the model succeeds.
We know what happens when it lies about JSON.
We know what happens when it calls the wrong tool.
We know what happens when it times out halfway through a stream.
We know what happens when it is rate-limited twice and succeeds once.

That confidence does not come from repeatedly asking a production model to behave.

It comes from controlling the boundary, simulating failure deliberately and reserving real-model calls for the questions only real models can answer.

Mock the uncertainty you do not intend to test

Measure the uncertainty you do