Evaluations: measuring agent quality

Once an agent handles dozens of conversations a day, you can't read them all. Evaluations grade them automatically. Find them under Performance > Evaluations.

You define:

• Criteria: yes/no quality rules, such as "answer is grounded in a cited document" or "no discount offered without approval". Each criterion has an action: Flag (mark for review) or Notify (alert the agent's managers right away by Slack or email).
• Tags: labels for sorting conversations, such as "billing" or "bug report". They don't affect the grade.
• Data points: values pulled out of each conversation, such as "order number" or "refund requested: yes/no".
• Optional sentiment: positive, neutral or negative tone.

After a conversation completes, an evaluator model reads the full transcript and gives a grade: Pass, Flag or Notify, with its reasoning per criterion. If the conversation continues and completes again, it is re-evaluated. Incognito chats are never evaluated.

You can also run evaluations by hand, on older conversations, or after changing criteria. With the Manage Evaluations ability, editors can ask the agent itself to review failures and propose new criteria; changes need approval.

Good practice:

• Start with 3 to 5 criteria tied to real risks.
• Read flagged conversations every week and fix instructions or skills.
• Remember the evaluator is also an AI and can be wrong; spot-check its judgements.

Evaluations use credits, depending on transcript length and the number of criteria. Gumloop's Reflections feature also flags wasted credits.