What you'll learn

Gain an in-depth understanding of the architecture behind embedding models; and learn how to train and use them.
Learn how to use different embedding models such as Word2Vec and BERT in various semantic search systems.
Learn how to build and train dual encoder models using contrastive loss, enhancing the accuracy of question-answer retrieval applications.

Course recap

PRO

This technical course, built in partnership with Vectara, provides a deep dive into how embedding models work -- from word embeddings (Word2Vec, GloVe) through contextualized token embeddings (BERT) to sentence embedding models based on dual encoder architectures. The course focuses on building blocks rather than applications.

Concept map

Key concepts

EmbeddingsVector representations that capture semantic meaning of words, sentences, or other data
Contextualized embeddingsBERT-style embeddings where the same word gets different vectors depending on surrounding context (e.g., "bat" in baseball vs. animal contexts)
Dual encoder architectureTwo separate encoders (question encoder and answer encoder) trained with contrastive loss for retrieval tasks
Contrastive lossTraining objective that pushes similar pairs closer and dissimilar pairs farther apart in embedding space, implemented via cross-entropy loss trick
Retrieval-Augmented GenerationUsing embedding-based retrieval to find relevant context for LLM generation
Cross encoderBERT-based classifier that determines semantic similarity by processing concatenated sentence pairs; more accurate but slower than dual encoders
Two-stage retrieval (Retrieve and Rerank)Using fast embedding-based retrieval as first-line filter, then cross-encoder reranker for top results

Lesson highlights

1.Introduction — Course overview, history of embeddings from Word2Vec to modern models
2.Introduction to embedding models — Vector embeddings, applications in NLP, sentence embeddings in RAG
3.Contextualized token embeddings — Word embeddings limitations, transformer architecture, BERT encoder, cross encoders
4.Token vs. sentence embedding — Why mean pooling fails, STS benchmark evaluation, dual encoder motivation
5.Training a dual encoder — Building encoders in PyTorch, contrastive loss with cross-entropy trick, training loop
6.Using embeddings in RAG — Question/answer encoders in production, ANN algorithms (HNSW, FAISS), DPR model
7.Conclusion — Two-stage retrieval pipeline, hybrid search, MMR for diversity

About this course

Join our new short course, Embedding Models: From Architecture to Implementation! Learn from Ofer Mendelevitch, Head of Developer Relations at Vectara.

This course goes into the details of the architecture and capabilities of embedding models, which are used in many AI applications to capture the meaning of words and sentences.

You will learn about the evolution of embedding models, from word to sentence embeddings, and build and train a simple dual encoder model. This hands-on approach will help you understand the technical concepts behind embedding models and how to use them effectively.

In detail, you’ll:

Learn about word embedding, sentence embedding, and cross-encoder models; and how they can be used in RAG
Understand how transformer models, specifically BERT (Bi-directional Encoder Representations from Transformers), are trained and used in semantic search systems
Gain knowledge of the evolution of sentence embedding and understand how the dual encoder architecture was formed
Use a contrastive loss to train a dual encoder model, with one encoder trained for questions and another for the responses
Utilize separate encoders for question and answer in a RAG pipeline and see how it affects the retrieval compared to using a single encoder model.

By the end of this course, you will understand word, sentence, and cross-encoder embedding models, and how transformer-based models like BERT are trained and used in semantic search. You will also learn how to train dual encoder models with contrastive loss and evaluate their impact on retrieval in a RAG pipeline.

Who should join?

This course is ideal for data scientists, machine learning engineers, NLP enthusiasts, and anyone who wants to learn about the creation and implementation of embedding models, which are crucial for building semantic retrieval systems. Whether you’re familiar with generative AI applications or new to the concept, if you have basic Python knowledge, this course offers a deep dive into how these models are built and capture the semantic meaning of words and sentences.

Embedding Models: from Architecture to Implementation

What you'll learn

Course recap

Concept map

Key concepts

Lesson highlights

About this course

Who should join?

Course Outline

Elevate your learning experience with Pro

Instructor

Ofer Mendelevitch

Embedding Models: from Architecture to Implementation

Want to learn more about Generative AI?