TopolearnNon-deterministic LLM apps need deterministic tests for logic and systematic evaluation for quality.
Separate two concerns.
Logic tests should be deterministic and offline. Replace the model with a fake from langchain_core.language_models.fake_chat_models:
GenericFakeChatModel(messages=iter([...]))returns scriptedAIMessages (including ones withtool_calls) and supports streaming.FakeListChatModel(responses=[...])returns strings in order.DeterministicFakeEmbeddingfor vector stores.
This lets you test routing in graphs, tool execution, parsing and state updates with pytest. Fakes that must support bind_tools can subclass GenericFakeChatModel and return self.
Observability: set LANGSMITH_TRACING=true and LANGSMITH_API_KEY to send traces of every chain, model and tool call to LangSmith, including inputs, outputs, latency and token usage. Without LangSmith, pass callbacks=[handler] in the config, where handler subclasses BaseCallbackHandler (on_llm_end, on_tool_start...), or use OpenTelemetry-based tools such as MLflow tracing.
Quality evaluation: build a dataset of inputs and reference outputs, run the app over it, and score with code checks and LLM-as-judge evaluators (LangSmith evaluate, or other frameworks). Track scores per version of prompts and models, and add failing production traces to the dataset.
Snapshot tests on exact LLM wording are brittle; assert on structure, tool calls and key facts instead.
Example
from langchain.agents import create_agent
from langchain_core.language_models.fake_chat_models import GenericFakeChatModel
from langchain_core.messages import AIMessage
from langchain_core.tools import tool
class FakeToolModel(GenericFakeChatModel):
def bind_tools(self, tools, **kwargs): # fakes don't bind tools by default
return self
@tool
def get_price(item: str) -> str:
"""Get the price of an item."""
return {"apple": "$1"}.get(item, "unknown")
def test_agent_calls_price_tool():
model = FakeToolModel(messages=iter([
AIMessage("", tool_calls=[{"name": "get_price", "args": {"item": "apple"}, "id": "c1"}]),
AIMessage("An apple costs $1."),
]))
agent = create_agent(model, tools=[get_price])
out = agent.invoke({"messages": [("user", "Price of an apple?")]})
assert [m.type for m in out["messages"]] == ["human", "ai", "tool", "ai"]
assert out["messages"][2].content == "$1"
test_agent_calls_price_tool()
print("agent logic test passed") # run with pytest in a real projectRemember
- Use fake chat models and embeddings for fast, deterministic logic tests
- Trace with LangSmith (env vars) or callback handlers / OpenTelemetry
- Evaluate quality on datasets with code checks and LLM judges; avoid exact-wording asserts
Watch out for
- Testing agents only end-to-end against a live model, making tests slow, flaky and costly.