Beginner2h13m

Open Source Models with Hugging Face

Instructors: Maria Khalusova, Marc Sun, Younes Belkada

Earn an accomplishment with PRO · Updated Oct 7, 2025

Enroll for Free

Beginner
2h13m
16 Video Lessons
13 Code Examples
1 Graded Assignment PRO
Earn an accomplishment with PRO
Updated Oct 7, 2025
Instructors: Maria Khalusova, Marc Sun, Younes Belkada
Hugging Face
Learn more aboutMembership PRO Plan

Enroll for Free

What you'll learn

Find and filter open source models on Hugging Face Hub based on task, rankings, and memory requirements.
Write just a few lines of code using the transformers library to perform text, audio, image, and multimodal tasks.
Easily share your AI apps with a user-friendly interface or via API and run them on the cloud using Gradio and Hugging Face Spaces.

Course recap

PRO

This course teaches how to navigate the Hugging Face Hub to find, load, and combine open-source models for text, audio, image, and multimodal tasks. Built in partnership with Hugging Face, it covers the full spectrum of model types available in the ecosystem and demonstrates how to string them together into practical applications.

Concept map

Key concepts

TransformersThe core architecture behind modern NLP, vision, and audio models; the course uses the Hugging Face Transformers library extensively
EmbeddingsSentence embeddings for measuring semantic similarity using Sentence Transformers and cosine distance
Pipeline APIHigh-level abstraction from Transformers that handles preprocessing (tokenization, spectrogram conversion, image normalization) automatically
Multimodal modelsModels like BLIP and CLIP that accept multiple input types (text + image) for tasks like VQA, image-text matching, and zero-shot classification
Fine-TuningReferenced as the basis for model specialization; distilled models like Distil-Whisper trade small accuracy losses for large speed gains
Sampling rateCritical concept for audio models; mismatch between training and inference sampling rates causes incorrect transcription lengths
Model cardsDocumentation on the Hub describing architecture, training data, limitations, and licensing

Lesson highlights

1.Introduction — Overview of course goals and open-source AI paradigm
2.Selecting models — Navigating Hugging Face Hub, filtering by task/language/license, estimating memory requirements
3.NLP (chatbot) — Building a chatbot with BlenderBot, conversation memory, Open LLM Leaderboard
4.Translation and Summarization — NLLB for 200-language translation, BART-Large-CNN for summarization, BFloat16 compression
5.Sentence Embeddings — Measuring sentence similarity with MiniLM, cosine similarity matrices
6.Zero-Shot Audio Classification — CLAP model for classifying arbitrary sounds without fine-tuning, sampling rate fundamentals
7.Automatic Speech Recognition — Whisper model, handling long audio with chunking and batch processing
8.Text to Speech — VITS model from Kakao Enterprise for text-to-audio generation
9.Object Detection — DETR-ResNet50 for detecting objects, building Gradio demos
10.Image Segmentation — Segment Anything Model (SAM) for mask generation, visual prompting with 2D points, depth estimation with DPT
11.Image Retrieval — BLIP for image-text matching using softmax over similarity logits
12.Image Captioning — BLIP for conditional and unconditional image captioning
13.Multimodal Visual Question Answering — BLIP for answering questions about images
14.Zero-Shot Image Classification — CLIP for classifying images against arbitrary text labels
15.Deployment — Deploying models to Hugging Face Spaces, using Gradio Client as API, Zero GPU feature
16.Conclusion — Recap and encouragement to build creative applications

About this course

The availability of models and their weights for anyone to download enables a broader range of developers to innovate and create.

In this course, you’ll select open source models from Hugging Face Hub to perform NLP, audio, image and multimodal tasks using the Hugging Face transformers library. Easily package your code into a user-friendly app that you can run on the cloud using Gradio and Hugging Face Spaces.

You will:

Use the transformers library to turn a small language model into a chatbot capable of multi-turn conversations to answer follow-up questions.
Translate between languages, summarize documents, and measure the similarity between two pieces of text, which can be used for search and retrieval.
Convert audio to text with Automatic Speech Recognition (ASR), and convert text to audio using Text to Speech (TTS).
Perform zero-shot audio classification, to classify audio without fine-tuning the model.
Generate an audio narration describing an image by combining object detection and text-to-speech models.
Identify objects or regions in an image by prompting a zero-shot image segmentation model with points to identify the object that you want to select.
Implement visual question answering, image search, image captioning and other multimodal tasks.
Share your AI app using Gradio and Hugging Face Spaces to run your applications in a user-friendly interface on the cloud or as an API.

The course will provide you with the building blocks that you can combine into a pipeline to build your AI-enabled applications!