Intermediate1h

Evaluating and Debugging Generative AI

Instructor: Carey Phelps

Earn an accomplishment with PRO · Updated May 14, 2026

Enroll for Free

Intermediate
1h
7 Video Lessons
5 Code Examples
1 Graded Assignment PRO
Earn an accomplishment with PRO
Updated May 14, 2026
Instructor: Carey Phelps
Weights and Biases
Learn more aboutMembership PRO Plan

Enroll for Free

What you'll learn

Learn to evaluate programs utilizing LLMs as well as generative image models using platform-independent tools
Instrument a training notebook, and add tracking, versioning, and logging
Implement monitoring and tracing of LLMs over time in complex interactions

Course recap

PRO

This course, taught by Carey Phelps of Weights & Biases, covers tools and best practices for systematically tracking and debugging generative AI models during development. It addresses both LLMs (text generation) and diffusion models (image generation), emphasizing that generative models have complex outputs that are harder to evaluate than traditional supervised learning.

Concept map

Key concepts

Experiment trackingSystematically logging metrics, hyperparameters, code state, and git commits for reproducibility
Fine-TuningFine-tuning language models (TinyStories on D&D character backstories) with qualitative evaluation of generated samples
Diffusion modelsDenoising models trained to remove noise from images; loss curves flatten early while sample quality continues improving
LLM tracingRecording inputs, outputs, and execution hierarchy of LLM chains and agents for debugging
ArtifactsVersioned storage for datasets, models, and predictions within ML experiment runs
Model RegistryCentral system for managing model lifecycle from staging to production

Lesson highlights

1.Introduction — Course overview and W&B tool suite
2.Instrument W&B — Adding experiment tracking to training code with wandb.init, wandb.log
3.Training a Diffusion Model with W&B — Training sprite generation diffusion model, artifact versioning, model registry
4.Evaluating Diffusion Models — Comparing DDPM vs DDIM samplers visually using W&B Tables
5.LLM Evaluation and Tracing with W&B — OpenAI API evaluation with Tables, custom chain tracing, LangChain agent tracing
6.Fine-tuning a Language Model — Fine-tuning TinyStories with HuggingFace Trainer, qualitative evaluation of generations
7.Conclusion — Summary of iterative development workflow

About this course

Machine learning and AI projects require managing diverse data sources, vast data volumes, model and parameter development, and conducting numerous test and evaluation experiments. Overseeing and tracking these aspects of a program can quickly become an overwhelming task.

This course will introduce you to Machine Learning Operations tools that manage this workload. You will learn to use the Weights & Biases platform which makes it easy to track your experiments, run and version your data, and collaborate with your team.

This course will teach you to:

Instrument a Jupyter notebook
Manage hyperparameter config
Log run metrics
Collect artifacts for dataset and model versioning
Log experiment results
Trace prompts and responses to LLMs over time in complex interactions

When you complete this course, you will have a systematic workflow at your disposal to boost your productivity and accelerate your journey toward breakthrough results.

Who should join?

Anyone who has familiarity with Python and PyTorch or similar framework and an interest in managing, versioning, and debugging their machine learning workflow.