TopolearnWithout an eval you can't tell whether a new prompt, model or effort level made your app better or worse.
An evaluation (eval) is a repeatable test for an AI feature: a fixed set of inputs, what a good answer looks like, and an automatic way to score each output. You run it every time you change a prompt, a model or a setting, and compare scores.
Start by writing down the expected behaviour in measurable terms ("the category is correct for at least 95 percent of tickets"). Then collect test cases that look like real traffic, with human-checked correct answers, plus awkward ones: empty input, very long input, off-topic or hostile messages.
Choose the simplest grader that works:
- Code checks: exact match, "contains", valid JSON. Fast and reliable whenever the answer is checkable.
- Model graders: a second model scores the output against a rubric and returns a constrained verdict, useful for tone or completeness. Check it agrees with human judgement on a sample before trusting it.
- Human review: best quality, slowest; use for spot checks.
Look at the failures, not just the average score, and keep the test set under version control with your code.
Tooling note for late 2026: OpenAI is deprecating its hosted Evals platform. Existing evals become read-only on 31 October 2026 and the platform shuts down on 30 November 2026. OpenAI points users to the open-source tool Promptfoo, and a plain Python script like the example works with any provider. Flex processing or the Batch API make large eval runs cheaper.
Example
from typing import Literal
from pydantic import BaseModel
from openai import OpenAI
client = OpenAI()
CASES = [("I was charged twice", "billing"), ("App crashes on login", "bug"),
("", "other"), ("Can I get a refund?", "billing")]
class Label(BaseModel):
category: Literal["billing", "bug", "other"]
def classify(text: str, model: str) -> str:
r = client.responses.parse(model=model, text_format=Label,
instructions="Classify the support ticket. Empty or unclear -> other.",
input=text or "(empty)")
return r.output_parsed.category
for model in ["gpt-6-luna", "gpt-6-astra"]:
got = [(t, want, classify(t, model)) for t, want in CASES]
score = sum(w == g for _, w, g in got) / len(got)
print(model, f"{score:.0%}", [t for t, w, g in got if w != g])Remember
- Measurable criteria + representative test set + automatic grading
- Prefer code graders; validate model graders against humans
- OpenAI's hosted Evals platform shuts down 30 Nov 2026; plain scripts or Promptfoo work
Watch out for
- Choosing a cheaper model after eyeballing three examples instead of running the full test set.