TopolearnWithout an eval you cannot tell whether a new prompt or a cheaper model made things better or worse.
Changing a prompt or switching models feels like an improvement until a user finds the case you broke. An evaluation (eval) is a repeatable test: a fixed set of inputs, the answers you expect or the qualities you want, and a way to score every output.
Start with success criteria that are specific and measurable: "at least 95 percent of tickets get the right category" rather than "works well". Most apps need several: accuracy, tone, format, latency and cost.
Then build a test set that looks like real traffic, including awkward cases: empty input, very long input, off-topic or hostile messages, ambiguous requests. More cases with automatic grading beat a few hand-checked ones. Claude itself can help draft extra cases from a few real ones.
Pick the simplest grader that works:
- Code-based: exact match, "contains", valid JSON, a regex. Fast and reliable; use whenever the answer is checkable.
- Model-based ("LLM as judge"): a second model scores the output against a clear rubric and returns a constrained verdict such as
correct/incorrect. Good for tone or completeness. Check it agrees with your own judgement on a sample first, and ideally use a different model from the one being tested. - Human review: best quality, slowest; keep it for spot checks.
Run the eval on every prompt or model change, track the score over time, and look at failures, not just the average. Structured outputs make model-based grading far more reliable.
Example
import anthropic
from typing import Literal
from pydantic import BaseModel
client = anthropic.Anthropic()
CASES = [("I was charged twice", "billing"), ("App crashes on login", "bug"),
("", "other"), ("Can I get a refund?", "billing")]
class Label(BaseModel):
category: Literal["billing", "bug", "other"]
def classify(text: str) -> str:
r = client.messages.parse(
model="claude-haiku-5-5", max_tokens=50, output_format=Label,
system="Classify the support ticket. Empty or unclear -> other.",
messages=[{"role": "user", "content": text or "(empty)"}])
return r.parsed_output.category
results = [(t, want, classify(t)) for t, want in CASES]
score = sum(want == got for _, want, got in results) / len(results)
print(f"accuracy {score:.0%}")
for t, want, got in results:
if want != got:
print("FAIL:", repr(t), "expected", want, "got", got)Remember
- Write measurable success criteria before tuning
- Test sets should mirror real traffic and include edge cases
- Prefer code graders; validate any LLM judge against human judgement
Watch out for
- Judging a prompt change on three hand-picked examples.