Learn how LLM-based testing differs from traditional software testing and implement rules-based testing to assess your LLM application.
Intermediate1h13m
Automated Testing for LLMOps
Instructor: Rob Zuber
Earn an accomplishment with PRO · Updated Oct 7, 2025
- Intermediate
- 1h13m
- 6 Video Lessons
- 4 Code Examples
- 2 Graded Assignments PRO
- Earn an accomplishment with PRO
- Updated Oct 7, 2025
- Instructor: Rob Zuber
CircleCI- Learn more aboutMembership PRO Plan
What you'll learn
Build model-graded evaluations to test your LLM application using an evaluation LLM.
Automate your evals (rules-based and model-graded) using continuous integration tools from CircleCI.
Course recap
PROThis course teaches how to build automated testing pipelines for LLM applications, integrating evaluation into CI/CD workflows. Students learn to create rules-based evaluations for per-commit testing and model-graded evaluations for pre-release testing, using CircleCI as the CI/CD platform.
Concept map
Key concepts
- Rules-based evalsAutomated checks using code to verify LLM output follows guidelines (format, content restrictions, factual accuracy)
- Model-graded evalsUsing an LLM to evaluate another LLM's output, especially for subjective quality assessment
- Per-commit testingLightweight rules-based evals that run on every code change
- Pre-release testingMore robust model-graded evals that run before deployment
- CI/CD for LLM appsIntegrating LLM evaluation into continuous integration pipelines
Lesson highlights
- 1.Introduction — LLMOps testing overview
- 2.Rules-Based Evals — Format validation, guideline adherence checks
- 3.Automating Model-Graded Evals — LLM-as-judge for output format validation
- 4.Comprehensive Evaluation — Hallucination detection, multi-data-point testing, human review
- 5.CI/CD Integration — CircleCI pipeline configuration, per-commit and pre-release workflows
- 6.Conclusion
About this course
In this course, you will learn how to create a continuous integration (CI) workflow to evaluate your LLM applications at every change for faster, safer, and more efficient application development.
When building applications with generative AI, model behavior is less predictable than traditional software. That’s why systematic testing can make an even bigger difference in saving you development time and cost.
Continuous integration, a key part of LLMOps, is the practice of making small changes to software in development and thoroughly testing them to catch issues early when they are easier to fix. With a robust automated testing pipeline, you’ll be able to isolate bugs before they accumulate – when they’re easier and less costly to fix. Automated testing lets your team focus on building new features, so that you can iterate and ship products faster.
After completing this course, you will be able to:
- Write robust LLM evaluations to cover common problems like hallucinations, data drift, and harmful or offensive output.
- Build a continuous integration (CI) workflow to automatically evaluate every change to your application.
- Orchestrate your CI workflow to run specific evaluations at different stages of development.
Who should join?
Anyone with basic Python knowledge and familiarity with building LLM-based applications.
Course Outline
6 Lessons・4 Code Examples- IntroductionVideo・3m
- Introduction to Continuous Integration (CI)Video・4m
- Overview of Automated EvalsVideo with Code Example・23m
- Automating Model-Graded EvalsVideo with Code Example・7m
- Comprehensive Testing FrameworkVideo with Code Example・12m
- ConclusionVideo・1m
- Quiz
Graded・Quiz
・10m - Quiz
Graded・Quiz
・10m - Optional: Exploring the CircleCI config fileCode Example・1m
Elevate your learning experience with Pro
Upgrade to Pro and gain unlimited accomplishments on your resume
Instructor
Course access is free for a limited time during the DeepLearning.AI learning platform beta!
Want to learn more about Generative AI?
Keep learning with updates on curated AI news, courses, and events, as well as Andrew’s thoughts from DeepLearning.AI!