TopolearnWithout systematic evaluation, prompt and model changes to LLM apps are guesswork that regress silently.
mlflow.genai.evaluate runs your application over an evaluation dataset, traces each call, and applies scorers to the traces:
mlflow.genai.evaluate(data=eval_data, predict_fn=my_app, scorers=[...])- data: a list of dicts or DataFrame with
inputs(keyword arguments forpredict_fn) and optionalexpectations(ground truth such asexpected_responseorexpected_facts). It can also be an MLflow evaluation dataset or existing traces, so production traffic can be scored after the fact. - predict_fn: your app; every call is traced.
- scorers: built-in LLM judges such as
Correctness,RelevanceToQuery,RetrievalGroundedness,SafetyandGuidelines(name=..., guidelines="..."), or custom ones.
Custom scorers are functions decorated with @scorer that receive inputs, outputs, expectations and/or trace, and return a bool, number, string, or Feedback with a rationale. Code-based scorers are cheap and deterministic, good for formats, lengths and exact checks; judges handle semantic quality. make_judge creates a custom LLM judge from natural-language instructions.
Results are logged to a run, with per-row assessments attached to traces, and the UI compares runs side by side. Judges themselves need validation: compare their verdicts with human labels on a sample before trusting them as a release gate.
Registered prompts (mlflow.genai.register_prompt) and set_active_model complete the loop: change a prompt, evaluate, compare versions.
Example
import mlflow
from mlflow.genai.scorers import scorer
def my_app(question: str) -> str: # stand-in for an LLM app
return "Paris is the capital of France." if "France" in question else "I am not sure."
@scorer
def mentions_expected(outputs, expectations) -> bool:
return expectations["expected_response"].lower() in outputs.lower()
@scorer
def concise(outputs) -> bool:
return len(outputs.split()) <= 12
data = [
{"inputs": {"question": "Capital of France?"}, "expectations": {"expected_response": "Paris"}},
{"inputs": {"question": "Capital of Peru?"}, "expectations": {"expected_response": "Lima"}},
]
result = mlflow.genai.evaluate(data=data, predict_fn=my_app,
scorers=[mentions_expected, concise])
print(result.metrics)
# {'mentions_expected/mean': 0.5, 'concise/mean': 1.0}
# Built-in judges (need an LLM endpoint): Correctness(), Guidelines(name=..., guidelines=...)Remember
mlflow.genai.evaluate= dataset + traced predict_fn + scorers- Combine cheap code scorers with LLM judges for semantic quality
- Validate judges against human labels before using them as gates
Watch out for
- Trusting an LLM judge's scores without checking agreement with human reviewers on a sample.