Gain in-depth knowledge of the steps to pretrain an LLM, encompassing all the steps, from data preparation, to model configuration and performance assessment.
Intermediate1h29m
Pretraining LLMs
Instructors: Sung Kim, Lucy Park
Earn an accomplishment with PRO · Updated Oct 7, 2025
- Intermediate
- 1h29m
- 8 Video Lessons
- 6 Code Examples
- 1 Graded Assignment PRO
- Earn an accomplishment with PRO
- Updated Oct 7, 2025
- Instructors: Sung Kim, Lucy Park
Upstage- Learn more aboutMembership PRO Plan
What you'll learn
Explore various options for configuring your model’s architecture, including modifying Meta’s Llama models to create larger or smaller versions and initializing weights either randomly or from other models.
Learn innovative pretraining techniques like Depth Upscaling, which can reduce training costs by up to 70%.
Course recap
PROThis course, taught by the Upstage team, covers the complete process of pretraining a large language model from scratch. It walks through data collection and cleaning, model initialization strategies (random weights, continued pretraining, upscaling, downscaling), tokenizer training, data packaging for efficient training, and the actual training loop with distributed computing.
Concept map
Key concepts
- Decoder-only architectureAutoregressive transformer architecture (GPT, Llama, Mistral) used for next-token prediction
- Data qualityDeduplication, language filtering, quality filtering, PII removal, and toxicity filtering as essential preprocessing steps
- Model upscalingDuplicating layers from a smaller model to create a larger one, then continuing pretraining to achieve coherence
- Model downscalingRemoving middle layers from a larger model to create a smaller one
- Continued pretrainingStarting from existing model weights and training on new domain data; much faster than training from scratch
- Tokenizer trainingBPE tokenization on domain-specific data to improve encoding efficiency
- Distributed trainingUsing FSDP (Fully Sharded Data Parallelism) to split model across multiple GPUs
Lesson highlights
- 1.Introduction — Why pretrain, overview of the pretraining landscape
- 2.Data Preparation — Web scraping, Hugging Face datasets, deduplication, language filtering with FastText, data cleaning pipeline
- 3.Model Initialization — Random weights, continued pretraining, upscaling and downscaling techniques, Solar model case study
- 4.Data Packaging — Tokenization, concatenation, chunking into fixed-length sequences for efficient training
- 5.Tokenizer Training — BPE algorithm, training custom tokenizers on domain data
- 6.Training the Model — Training loop, loss computation, distributed training with FSDP
- 7.Model Evaluation — Perplexity, benchmark evaluation, human evaluation
- 8.Conclusion
About this course
In Pretraining LLMs you’ll explore the first step of training large language models using a technique called pretraining. You’ll learn the essential steps to pretrain an LLM, understand the associated costs, and discover how starting with smaller, existing open source models can be more cost-effective.
Pretraining involves teaching an LLM to predict the next token using vast text datasets, resulting in a base model, and this base model requires further fine-tuning for optimal performance and safety. In this course, you’ll learn to pretrain a model from scratch and also to take a model that’s already been pretrained and continue the pretraining process on your own data.
In detail:
- Explore scenarios where pretraining is the optimal choice for model performance. Compare text generation across different versions of the same model to understand the performance differences between base, fine-tuned, and specialized pre-trained models.
- Learn how to create a high-quality training dataset using web text and existing datasets, which is crucial for effective model pretraining.
- Prepare your cleaned dataset for training. Learn how to package your training data for use with the Hugging Face library.
- Explore ways to configure and initialize a model for training and see how these choices impact the speed of pretraining.
- Learn how to configure and execute a training run, enabling you to train your own model.
- Learn how to assess your trained model’s performance and explore common evaluation strategies for LLMs, including important benchmark tasks used to compare different models’ performance.
After taking this course, you’ll be equipped with the skills to pretrain a model—from data preparation and model configuration to performance evaluation.
Who should join?
This course is ideal for AI enthusiasts, data scientists, and machine learning engineers who want to learn the complete process of pretraining LLMs. Basic knowledge of Python and large language models is recommended.
Course Outline
8 Lessons・6 Code Examples- IntroductionVideo・6m
- Why Pre-trainingVideo with Code Example・12m
- Data PreparationVideo with Code Example・16m
- Packaging Data for PretrainingVideo with Code Example・8m
- Model InitializationVideo with Code Example・16m
- Training in ActionVideo with Code Example・11m
- EvaluationVideo with Code Example・7m
- ConclusionVideo・1m
- Quiz
Graded・Quiz
・10m
Elevate your learning experience with Pro
Upgrade to Pro and gain unlimited accomplishments on your resume
Instructors
Course access is free for a limited time during the DeepLearning.AI learning platform beta!
Want to learn more about Generative AI?
Keep learning with updates on curated AI news, courses, and events, as well as Andrew’s thoughts from DeepLearning.AI!