Learn how the Jamba model integrates transformers and the Mamba architecture to efficiently process long contexts while maintaining quality.
Beginner1h14m
Build Long-Context AI Apps with Jamba
Instructors: Chen Wang, Chen Almagor
Earn an accomplishment with PRO · Updated Oct 7, 2025
- Beginner
- 1h14m
- 9 Video Lessons
- 4 Code Examples
- 1 Graded Assignment PRO
- Earn an accomplishment with PRO
- Updated Oct 7, 2025
- Instructors: Chen Wang, Chen Almagor
AI21 labs- Learn more aboutMembership PRO Plan
What you'll learn
Understand the training process for long context models, and the metrics used to evaluate their performance.
Gain hands-on experience applying Jamba to tasks such as processing large documents, tool-calling, and building large context RAG apps.
Course recap
PROBuilt in partnership with AI21 Labs, this course explores the Jamba model -- a novel hybrid architecture combining traditional transformers with Mamba (state space models) to efficiently handle very long input contexts. While transformers excel at attention-based relationships but scale quadratically, Mamba offers linear-time processing but struggles with long-distance relationships. Jamba captures the best of both worlds.
Concept map
Key concepts
- Jamba architectureHybrid transformer-Mamba model that combines attention (for long-distance relationships) with state space models (for efficiency)
- Attention MechanismTransformer attention compares every pair of tokens but has quadratic cost in input length
- Mamba / State Space ModelsLinear-time sequence modeling that compresses input to fixed-size representation but can lose information
- Long context windowsExtended input capacity enabled by the hybrid architecture
- Context length evaluationMethods for assessing model performance as context length grows
Lesson highlights
- 1.Introduction — Transformer limitations, Mamba, and the Jamba hybrid
- 2.Jamba Architecture Overview — How transformer and Mamba layers are interleaved
- 3.State Space Models — Mamba's selective state space mechanism
- 4.Context Length Strategies — Techniques for expanding model context
- 5.Evaluating Long Context — Benchmarks and evaluation methods
- 6.Prompting with Jamba — Leveraging long context for better prompts
- 7.Document Processing — Processing large documents with extended context
- 8.Tool Calling and RAG — Combining long context with retrieval and tools
- 9.Conclusion
About this course
Learn to use the Jamba model, a hybrid transformer-Mamba architecture trained to handle long contexts, in this new course, Build Long-Context AI Apps with Jamba, built in partnership with AI21 Labs and taught by Chen Wang and Chen Almagor.
The transformer architecture is the foundation of most large language models but it is computationally expensive when handling very long input contexts.
There’s an alternative to transformers called Mamba. Mamba is a selective state space model that can process very long contexts with a much lower computational cost. However, researchers found that the pure Mamba architecture underperforms in understanding the context, and the quality of the output can be lower even in tasks as simple as repeating the input in the output of the model.
To overcome these challenges, AI21 Labs developed the Jamba model, which combines Mamba’s computational efficiency with the transformer’s attention mechanism to help with the output quality.
In this course, you’ll learn about the Jamba architecture, how it works, and how it is trained. You’ll also learn how to prompt Jamba and use it to process long documents and build long-context RAG apps.
In detail, you’ll:
- Learn key information about Jamba’s large context model and its unique, hybrid architecture that allows for high-quality outputs, high throughput, and low memory usage.
- Learn how Jamba combines transformers with Mamba, a selective state-space model to achieve high performance and quality.
- Use the AI21SDK, especially its document parameter, with an example of prompting a large over 200k-token annual report.
- Use Jamba for tool-calling with hands-on examples from calling simple arithmetic functions to a function that returns SEC 10-Q company quarterly report.
- Learn how the training for long context is done, and the metrics used for evaluating the performance.
- Create a RAG app using the AI21 Conversational RAG tool and build your own RAG pipeline that uses the Jamba model with LangChain.
Start building AI apps that can handle context as long as all of your unread emails from the last 20 years!
Who should join?
Anyone who has basic Python knowledge and wants to learn more about how the Jamba model works and how it is used for building long-context AI apps.
Course Outline
9 Lessons・4 Code Examples- IntroductionVideo・3m
- OverviewVideo・5m
- Transformer-Mamba Hybrid LLM ArchitectureVideo・14m
- Jamba Prompting and DocumentsVideo with Code Example・8m
- Tool CallingVideo with Code Example・6m
- Expand the Context Window SizeVideo・13m
- Long Context PromptingVideo with Code Example・3m
- Conversational RAGVideo with Code Example・8m
- ConclusionVideo・1m
- Quiz
Graded・Quiz
・10m
Elevate your learning experience with Pro
Upgrade to Pro and gain unlimited accomplishments on your resume
Instructors
Course access is free for a limited time during the DeepLearning.AI learning platform beta!
Want to learn more about Generative AI?
Keep learning with updates on curated AI news, courses, and events, as well as Andrew’s thoughts from DeepLearning.AI!