What you'll learn

Understand development steps, from evaluation, through prompting, self-reflection, and fine-tuning, to improve your model’s reliability and accuracy.
Learn how memory tuning can increase your model performance by embedding facts into your model to reduce hallucination.
Use the Llama 3-8b model to build an LLM application that converts text to SQL with a custom schema.

Course recap

PRO

Built in partnership with Lamini and Meta, this course teaches a systematic framework for improving LLM application accuracy through three progressive steps: prompt engineering, evaluation dataset creation, and fine-tuning. Taught by Sharon Zhou (Lamini) and Amit Sangani (Meta Llama team), it uses a text-to-SQL example to demonstrate accuracy improvements from ~30% to 95%.

Concept map

Key concepts

Evaluation-driven improvementCreating rigorous evaluation datasets to measure performance before and after changes
Prompt EngineeringFirst line of improvement including self-reflection prompts and few-shot examples
Fine-TuningUsing LoRA/PEFT for parameter-efficient fine-tuning when prompt engineering reaches its limits
Memory TuningLamini's technique for achieving high factual accuracy (95%+) by teaching LLMs specific facts
Data amplificationUsing LLMs themselves to generate and expand training datasets when you think you don't have enough data
Text-to-SQLRunning example of converting natural language queries to SQL for a specific schema

Lesson highlights

1.Introduction — Framework overview: prompt engineer -> evaluate -> fine-tune
2.Building the LLM application — Setting up text-to-SQL pipeline with prompt engineering
3.Creating evaluation datasets — Measuring baseline performance rigorously
4.Prompt engineering and self-reflection — Improving accuracy through prompt iteration
5.Fine-tuning with LoRA — Parameter-efficient fine-tuning on small datasets (128 examples)
6.Memory Tuning — Advanced technique for factual accuracy at speed
7.Conclusion — Summary of the improvement framework

About this course

Join our new short course, Improving Accuracy of LLM Applications with AMD (formerly Lamini) and Meta. Learn from Sharon Zhou, Vice President of Artificial Intelligence, AMD, and Amit Sangani, Senior Director of Partner Engineering, Meta.

Many developers have experienced frustration with inconsistent results when working with LLM applications. This course offers a systematic approach to enhance the accuracy and reliability of your LLM applications.

You will build an SQL agent, add evaluation metrics to measure performance, and use prompt engineering and self-reflection to make the model perform better. Finally, you will fine-tune the model with techniques like LoRA and memory tuning that embeds facts in model weights to reduce hallucinations.

In this course, you’ll use Llama’s family of open-source models.

What you’ll do:

Build a text to SQL agent and simulate situations where it hallucinates to begin the evaluation process.
Build an evaluation framework to systematically measure performance, including criteria for good evaluations, best practices, and how to develop an evaluation score.
Learn how instruction fine-tuning enhances pre-trained LLMs to follow instructions, and how memory fine-tuning embeds facts to reduce hallucinations.
Break fine-tuning myths and see how Performance-Efficient Fine-tuning (PEFT) techniques like Low-Rank Adaptation(LoRA) reduce training time by 100x and Mixture of Memory Experts (MoME) reduces it even further.
Go through an iterative process of generating training data and fine-tuning, learning practical tips such as adding examples, generating variations, and filtering generated data to increase model accuracy.

Start improving the accuracy of LLM applications today!

Who should join?

This course is ideal for anyone with intermediate Python knowledge and familiarity with large language models (LLMs) looking to build more factual and precise LLM applications.

Improving Accuracy of LLM Applications

What you'll learn

Course recap

Concept map

Key concepts

Lesson highlights

About this course

Who should join?

Course Outline

Elevate your learning experience with Pro

Instructors

Sharon Zhou

Amit Sangani

Improving Accuracy of LLM Applications

Want to learn more about Generative AI?