TopolearnAgents are nondeterministic; you need repeatable measurements before and after every change.
Training (crewai train -n 3 -f trained.pkl) runs the crew several times and, after each task, asks you for feedback in the terminal ("too long", "always cite the source"). CrewAI turns that feedback into lasting suggestions, saved in a file (default trained_agents_data.pkl). Later runs load the suggestions and add them to the agents' prompts.
From Python: crew.train(n_iterations=3, filename="trained.pkl", inputs={...}). Training costs real model calls per iteration, so use a few representative inputs.
Testing (crewai test -n 3 -m <model>) runs the crew several times and has a model score each task from 1 to 10, then prints a table with averages and run times. In code: crew.test(n_iterations=3, eval_llm=..., inputs={...}).
Use these scores as a smoke alarm, not a verdict. An LLM judge has its own biases and can't know your business rules. For decisions that matter, build your own evaluation:
- A fixed set of test inputs with expected properties (facts that must appear, length limits, required fields).
- Code checks wherever possible, such as validating structured outputs or the presence of source links.
- An LLM judge with a precise rubric only for qualities code can't check.
- Run the set after every prompt, model or tool change and compare with the previous scores.
Because outputs vary from run to run, look at averages and failure rates over several iterations, never a single run. Track cost (token_usage) and time alongside quality: a small quality gain may not justify doubling cost.
Tracing on CrewAI AMP adds crewai eval for evaluating traced runs and comparing models.
Example
# Terminal, inside a crew project
crewai train -n 3 -f trained_agents_data.pkl # asks for your feedback after tasks
crewai test -n 3 -m gpt-4o-mini # scores each task 1-10, prints a table
crewai log-tasks-outputs # outputs of the latest kickoff
# The same from Python (e.g. in a notebook or CI script)
# crew.train(n_iterations=3, filename="trained_agents_data.pkl", inputs={"topic": "AI"})
# crew.test(n_iterations=3, eval_llm="openai/gpt-4o-mini", inputs={"topic": "AI"})Remember
crewai trainturns your feedback into saved suggestions for agentscrewai testruns several iterations and scores tasks with an LLM judge- Back judge scores with your own fixed test set and code checks
Watch out for
- Comparing two prompt versions on one run each; nondeterminism can make the worse one look better.