DeepLearning.AI
Intermediate1h43m

Reinforcement Fine-Tuning LLMs With GRPO

Instructors: Travis Addair, Arnav Garg

Predibase logo

Earn an accomplishment with PRO · Updated Oct 7, 2025

  • Intermediate
  • 1h43m
  • 10 Video Lessons
  • 7 Code Examples
  • 1 Graded Assignment PRO
  • Earn an accomplishment with PRO
  • Updated Oct 7, 2025
  • Instructors: Travis Addair, Arnav Garg
  • PredibasePredibase
  • Learn more aboutMembership PRO Plan

What you'll learn

  • Learn the foundations of reinforcement learning and how to use the Group Relative Policy Optimization (GRPO) algorithm to improve reasoning in large language models.

  • Design effective reward functions, and learn how rewards are converted into advantages to steer models toward high-quality behavior across multiple use cases.

  • Learn to use LLM as a Judge for subjective tasks, overcome reward hacking with penalty functions, and calculate the loss function in GRPO.

Course recap

PRO

This course uses the game of Wordle as a running example to teach Group Relative Policy Optimization (GRPO), a reinforcement learning algorithm for fine-tuning LLMs. Students learn to define programmatic reward functions, implement the GRPO training loop, and observe how RL fine-tuning teaches models strategic reasoning that supervised fine-tuning cannot achieve.

Concept map

Concepts in Reinforcement Fine-Tuning LLMs With GRPO and the courses that connect to themReinforcement Fine-Tuning LLMs with GRPOReinforcement Fine-Tuning…GRPO (Group Relative Policy Optimization)GRPO (Group Relative Poli…Reward functionsReinforcement Learning from Human FeedbackReinforcement Learning fr…LoRA (Low-Rank Adaptation)Chain-of-thought reasoningVerifiable rewardsPost-training of LLMsReinforcement Learning from Human FeedbackReinforcement Learning fr…Reasoning with o1Fine-Tuning and Reinforcement Learning for LLMs: Intro to Post-TrainingFine-Tuning and Reinforce…Generative AI with LLMsFinetuning Large Language ModelsFinetuning Large Language…Federated Fine-Tuning of LLMs with Private DataFederated Fine-Tuning of…Concepts in Reinforcement Fine-Tuning LLMs With GRPO and the courses that connect to themReinforcement Fine-Tuning LLMs with GRPOReinforcement F…GRPO (Group Relative Policy Optimization)GRPO (Group Rel…Reward functionsReinforcement Learning from Human FeedbackReinforcement L…LoRA (Low-Rank Adaptation)LoRA (Low-Rank…Chain-of-thought reasoningChain-of-though…Verifiable rewardsVerifiable rewa…

Key concepts

  • GRPO (Group Relative Policy Optimization)DeepSeek's RL algorithm: sample multiple responses per prompt, score with reward functions, push up above-average and down below-average responses
  • Reward functionsProgrammatic functions scoring model outputs (format checking, correctness verification, composite scoring)
  • Reinforcement Learning from Human FeedbackComparison of RLHF, DPO, and GRPO approaches; GRPO eliminates need for human preference labels
  • LoRA (Low-Rank Adaptation)Parameter-efficient fine-tuning used alongside GRPO to update only small adapter weights
  • Chain-of-thought reasoningModel emits reasoning in <think> tags before providing guesses; RL training improves reasoning quality over time
  • Verifiable rewardsTasks with objective correctness measures (Wordle letter feedback) where reward functions can be programmatically defined

Lesson highlights

  1. 1.Introduction — Overview of reinforcement fine-tuning and its advantages over SFT
  2. 2.Introduction to Reinforcement Learning — SFT vs. RLHF vs. DPO vs. GRPO comparison
  3. 3.Can an LLM Master Wordle — Base model vs. fine-tuned model gameplay comparison; Qwen 2.5 7B
  4. 4.Reward Functions for Wordle — Designing format, correctness, and composite reward functions
  5. 5.GRPO Algorithm Deep Dive — Group sampling, relative advantage computation, policy update
  6. 6.Training Loop Implementation — Setting up GRPO training with LoRA adapters
  7. 7.Evaluation — Comparing base and fine-tuned model performance across game simulations
  8. 8.Advanced Reward Design — Combining multiple reward signals with weighting
  9. 9.Scaling and Production — Considerations for larger models and production deployment
  10. 10.Conclusion

About this course

Join Reinforcement Fine-Tuning LLMs with GRPO, built in collaboration with Predibase, and taught by Travis Addair, its Co-Founder and CTO, and Arnav Garg, its Senior Engineer and Machine Learning Lead.

Reinforcement Fine-Tuning (RFT) is a technique for adapting LLMs to complex reasoning tasks like mathematics and coding. RFT leverages reinforcement learning (RL) to help models develop their strategies for completing a task, rather than relying on pre-existing examples as in traditional supervised fine-tuning. One RL algorithm, called Group Relative Policy Optimization (GRPO), is beneficial for tasks with verifiable outcomes and can work well even when you have fewer than 100 training examples. Using RFT to adapt small, open-source models can lead to competitive performance on reasoning tasks, giving you more options for your LLM-powered applications.

In this course, you’ll take a technical deep dive into RFT with GRPO. You’ll learn how to build reward functions that you can use in the GRPO training process to guide an LLM toward better performance on multi-step reasoning tasks.

In detail, you’ll learn:

  • When reinforcement fine-tuning is a better fit than supervised fine-tuning, especially for tasks involving multi-step reasoning or limited labeled data.
  • How GRPO uses programmable reward functions as a more scalable alternative to the human feedback required for other reinforcement learning algorithms, such as RLHF and DPO.
  • How to frame the Wordle game as a reinforcement fine-tuning problem and see how an LLM can learn to plan, analyze feedback, and improve its strategy over time.
  • How to design reward functions that power the reinforcement fine-tuning process.
  • Techniques for evaluating more subjective tasks, such as rating the quality of a text summary, using an LLM as a judge.
  • Why reward hacking happens and how to avoid it by adding penalty functions to discourage undesirable behaviors.
  • The four key components of the loss calculation in the GRPO algorithm: token probability distribution ratios, advantages, clipping, and KL-divergence.
  • How to launch reinforcement fine-tuning jobs using Predibase’s hosted training services.

By the end of this course, you’ll be able to fine-tune LLMs using reinforcement learning to improve reasoning without relying on large labeled datasets or subjective human feedback.

Who should join?

This course is for anyone who wants to fine-tune LLMs for complex reasoning tasks without relying on large labeled datasets. Ideal for those interested in reinforcement learning, LLM reasoning, and improving the performance of small, open-source models.

Course Outline

10 Lessons・7 Code Examples
Unlock certificates

Elevate your learning experience with Pro

Upgrade to Pro and gain unlimited accomplishments on your resume

Learn More

Instructors

Travis Addair

Travis Addair

Arnav Garg

Arnav Garg

Senior Machine Learning Engineer at Predibase

Course access is free for a limited time during the DeepLearning.AI learning platform beta!

Want to learn more about Generative AI?

Keep learning with updates on curated AI news, courses, and events, as well as Andrew’s thoughts from DeepLearning.AI!

Choose Your Plan

Planning for more users?
Learn More