Prompt vision models with text, coordinates, and bounding boxes, and tune hyper-parameters like guidance scale, strength, and number of inference steps.
Beginner1h32m
Prompt Engineering for Vision Models
Instructors: Abby Morgan, Jacques Verré, Caleb Kaiser
Earn an accomplishment with PRO · Updated Oct 7, 2025
- Beginner
- 1h32m
- 7 Video Lessons
- 4 Code Examples
- 1 Graded Assignment PRO
- Earn an accomplishment with PRO
- Updated Oct 7, 2025
- Instructors: Abby Morgan, Jacques Verré, Caleb Kaiser
Comet- Learn more aboutMembership PRO Plan
What you'll learn
Replace parts of an image with generated content with in-painting, a technique that combines object detection, image segmentation, and image generation.
Fine-tune a diffusion model to have even more control over your image generation to create specific images, including your own, rather than generically generated images.
Course recap
PROThis course teaches visual prompting techniques for computer vision models, focusing on Meta's Segment Anything Model (SAM) for segmentation and Stability AI's Stable Diffusion for image generation and inpainting. It covers prompting with pixel coordinates, bounding boxes, text, and images, and includes fine-tuning diffusion models with DreamBooth.
Concept map
Key concepts
- Visual promptingInteracting with pretrained vision models using inputs like coordinates, bounding boxes, embeddings, and text rather than retraining
- Segmentation masksPixel-level object identification using SAM prompted with points or bounding boxes
- InpaintingAI-powered image editing: replacing specific regions using a diffusion model guided by a mask and text prompt
- Diffusion modelsGenerate images by iteratively denoising Gaussian noise guided by text embeddings (U-Net + text encoder)
- Guidance scaleHyperparameter controlling how closely the diffusion model follows the text prompt; higher values increase fidelity but may reduce quality
- StrengthControls how much noise is added to the original image in image-to-image diffusion; determines edit intensity
- DreamBoothFine-tuning technique to teach a diffusion model new concepts from a few custom images
Lesson highlights
- 1.Overview — Prompt engineering fundamentals, visual prompting vs. traditional ML, experiment tracking with Comet
- 2.Segmentation with SAM — Prompting SAM with pixel coordinates and bounding boxes to generate masks
- 3.Text-based Segmentation — Using CLIP embeddings to prompt SAM with natural language
- 4.Image Generation — Stable Diffusion inpainting; guidance scale, strength, inference steps, negative prompts
- 5.Fine-tuning Diffusion Models — DreamBooth training to teach Stable Diffusion new visual concepts
- 6.Advanced Techniques — Combining segmentation and generation pipelines
- 7.Conclusion
About this course
Prompt engineering is used not only in text models but also in vision models. Depending on the vision model, they may use text prompts, but can also work with pixel coordinates, bounding boxes, or segmentation masks.
In this course, you’ll learn to prompt different vision models like Meta’s Segment Anything Model (SAM), a universal image segmentation model, OWL-ViT, a zero-shot object detection model, and Stable Diffusion 2.0, a widely used diffusion model. You’ll also use a fine-tuning technique called DreamBooth to tune a diffusion model to associate a text label with an object of your preference.
In detail, you’ll explore:
- Image Generation: Prompt with text and by adjusting hyperparameters like strength, guidance scale, and number of inference steps**.**
- Image Segmentation: Prompt with positive or negative coordinates, and with bounding box coordinates.
- Object detection: Prompt with natural language to produce a bounding box to isolate specific objects within images.
- In-painting: Combine the above techniques to replace objects within an image with generated content.
- Personalization with Fine-tuning: Generate custom images based on pictures of people or places that you provide, using a fine-tuning technique called DreamBooth.
- Iterating and Experiment Tracking: Prompting and hyperparameter tuning are iterative processes, and therefore experiment tracking can help to identify the most effective combinations. This course will use Comet, a library to track experiments and optimize visual prompt engineering workflows.
Who should join?
Prompt Engineering for Vision Models is a hands-on course that helps you get started with prompting vision models. Python experience is recommended.
Course Outline
7 Lessons・4 Code Examples
Elevate your learning experience with Pro
Upgrade to Pro and gain unlimited accomplishments on your resume
Instructors
Course access is free for a limited time during the DeepLearning.AI learning platform beta!
Want to learn more about Generative AI?
Keep learning with updates on curated AI news, courses, and events, as well as Andrew’s thoughts from DeepLearning.AI!