What you'll learn

Learn how to add observability to your agent to gain insights into its steps and know how to debug it.
Learn how to set up evaluations for the agent components by preparing testing examples, choosing the appropriate evaluator (code-based or LLM-as-a-Judge), and identifying the right metrics.
Learn how to structure your evaluation into experiments to iterate on and improve the output quality and the path taken by your agent.

Course recap

PRO

This course, built in partnership with Arize AI, teaches evaluation-driven development for AI agents. Students build a data analysis agent from scratch (with database lookup, analysis, and visualization tools) and learn to systematically evaluate every component: the router's tool selection, each skill's output quality, and the agent's trajectory efficiency.

Concept map

Key concepts

Evaluation-driven developmentUsing evaluations and measurements from your agent to guide where to spend improvement effort, creating a flywheel from development to production
AI AgentsSoftware systems with reasoning (LLM), routing (tool selection), and action (tool execution) components
ObservabilityComplete visibility into application layers via traces and spans using OpenTelemetry standards
LLM-as-a-judgeUsing a separate LLM to evaluate qualitative outputs like relevance, clarity, and correctness using discrete classification labels
Convergence scoreMetric measuring how often an agent takes the optimal path for a given set of inputs (0 to 1 scale)
Trajectory evaluationAssessing the sequence of steps an agent takes, measuring efficiency and detecting unnecessary loops

Lesson highlights

1.Introduction — Overview of evaluation-driven development for agents
2.Evaluation in the time of LLMs — How LLM evaluation differs from traditional software testing
3.Decomposing agents — Router, skills, and memory components; building the data analysis agent
4.Lab 1: Building your agent — Implementing the data analysis agent with three tools and OpenAI function calling
5.Tracing agents — Observability concepts, OpenTelemetry, traces and spans
6.Lab 2: Tracing your agent — Adding instrumentation with Arize Phoenix, manual and automatic tracing
7.Adding router and skill evaluations — Code-based evals, LLM-as-a-judge, human annotations
8.Lab 3: Adding router and skill evaluations — Implementing function calling evals, code runnability checks, SQL correctness
9.Adding trajectory evaluations — Agent trajectories, convergence measurement
10.Lab 4: Adding trajectory evaluations — Implementing convergence evaluations with Phoenix experiments
11.Adding structure to your evaluations — Evaluation-driven development cycle, structured experiments
12.Lab 5: Adding structure to your evaluations — Combining all evaluators into one experiment, iterating on agent performance
13.Improving your LLM-as-a-judge — Using experiments to improve evaluator accuracy, semantic similarity for comparison
14.Monitoring agents — Production monitoring, CI/CD integration, self-improving agents
15.Conclusion — Summary of techniques

About this course

Learn how to systematically assess and improve your AI agent’s performance in Evaluating AI Agents, a short course built in partnership with Arize AI and taught by John Gilhuly, Head of Developer Relations, and Aman Khan, Director of Product.

When you’re building an AI Agent, an important part of the development process is evaluations or evals. Whether you’re building a shopping assistant, coding agent, or research assistant, having a structured evaluation process helps you refine its performance systematically—rather than relying on trial and error.

With a systematic approach, you structure your evaluations to assess the performance of each component of the agent, as well as its end-to-end performance. For each component, you select the appropriate evaluators, testing examples, and metrics. This process helps you identify any areas of improvement so you can iterate on your agent during development and in production.

In this course, you’ll build an AI agent, add observability to visualize and debug its steps, and evaluate its performance component-wise.

In detail, you’ll:

Distinguish between evaluating LLM-based systems and traditional software testing.
Explore the basic structure of AI agents – routers, skills, and memory – and implement an AI agent from scratch.
Add observability to the agent by collecting traces of the steps taken by the agent and visualizing the traces.
Choose the appropriate evaluator – code-based, LLM-as-a-Judge, and human annotations – for each component of the agent.
Set up evaluations for the skills and router decisions of the agent example using code-based and LLM-as-a-judge evaluators, by creating testing examples from collected traces and preparing detailed prompts for the LLM-as-a-judge.
Compute a convergence score to evaluate if the example agent can respond to a query in an efficient number of steps.
Run structured experiments to improve the performance of the agent by exploring changes to the prompt, LLM model, or the agent’s logic.
Understand how to deploy these evaluation techniques to monitor the agent’s performance in production.

By the end of this course, you’ll know how to trace AI agents, systematically evaluate them, and improve their performance.

Who should join?

Anyone who has basic Python knowledge and wants to learn to evaluate, troubleshoot, and improve AI agents effectively—both during development and in production. Familiarity with prompting an LLM model would be helpful but not required.

Evaluating AI Agents

What you'll learn

Course recap

Concept map

Key concepts

Lesson highlights

About this course

Who should join?

Course Outline

Elevate your learning experience with Pro

Instructors

John Gilhuly

Aman Khan

Evaluating AI Agents

Want to learn more about Generative AI?