EE/CS 148B · Spring 2026

Large Language & Vision Models

This course introduces the foundations, design, training, and evaluation of large language models and vision-language models, with an emphasis on modern multimodal systems, reasoning, alignment, and emerging research directions.

Caltech Tue / Thu, 10.30am - 11.55am Chen 100

Course staff

Georgia

Instructor

Georgia Gkioxari

Ziqi

Teaching Assistant

Ziqi Ma

Damiano

Teaching Assistant

Damiano Marsili

Aadarsh

Teaching Assistant

Aadarsh Sahoo

Vansh

Teaching Assistant

Vansh Tibrewal

Vikrant

Teaching Assistant

Vikrant Malik

Sreeyutha

Teaching Assistant

Sreeyutha Ratala


Lecture schedule

Spring 2026 Schedule

Date Topic Notes
Lecture 1
Tue, Mar 31
 Course overview
  • ➤ why large language and vision models now?
  • ➤ the foundations of modern AI
  • ➤ course content, grading and policies
 syllabus (doc) · slides (pdf)
Lecture 2
Thu, Apr 2
 Transformer I: Self-Attention
  • ➤ Motivation of attention
  • ➤ Self-attention block
  • ➤ Transformer Encoder
 slides (pdf) · Assignment 1: Build an LM from Scratch (pdf)
Lecture 3
Tue, Apr 7
 Transformer II: Language Model
  • ➤ Cross-attention
  • ➤ Transformer Decoder
 slides (pdf)
Lecture 4
Thu, Apr 9
 LLM Post-training
  • ➤ Alignment & RLHF
  • ➤ Reasoning & GRPO
 slides - alignment (pdf) · slides - reasoning (pdf)
Lecture 5
Tue, Apr 14
 The Modern Transformer I
  • ➤ KV caching
  • ➤ Multi-Query Attention, Grouped-Query Attention
  • ➤ DeepSeek's Sparse Attention
  • ➤ Flash Attention
 slides (pdf)
Lecture 6
Thu, Apr 16
 The Modern Transformer II
  • ➤ RoPE
slides(pdf) · Assignment 2: Profiling & Reasoning (pdf)
Lecture 7
Tue, Apr 21
 The Modern Transformer III
  • ➤ Mixture of Experts (MoEs)
  • ➤ Motivation for MoEs
  • ➤ Implementation & Router Design
  • ➤ Shared, Sparse Experts
  • ➤ Training & Load Balancing
slides(pdf)
Lecture 8
Thu, Apr 23
 Vision Language Models
  • ➤ CLIP
slides(pdf)
Lecture 9
Tue, Apr 28
 Vision Language Models cont'd
  • ➤ VLM designs & recipes
slides(pdf)
Lecture 10
Thu, Apr 30
 Visual Representation Learning
  • ➤ Self-Supervised Learning
  • ➤ MAE
slides(pdf) · Assignment 3: Building a VLM from Scratch-ish(pdf)
Lecture 11
Tue, May 5
 Visual Representation Learning cont'd
  • ➤ SimCLR
  • ➤ DINO, DINOv2, DINOv3
slides(pdf)
Lecture 12
Thu, May 7
 NO CLASS
Lecture 13
Tue, May 12
 Generative Models I
  • ➤ Intro to Generative Models
  • ➤ Autoregressive Models
  • ➤ GPT, PixelCNN
slides(pdf)
Lecture 14
Thu, May 14
 Generative Models II
  • ➤ Autoencoders
  • ➤ Variational Autoencoders (VAE)
slides(pdf) · Assignment 4: Diffusion & Flow Matching(pdf)
Lecture 15
Tue, May 19
 Generative Models II cont'd
  • ➤ Autoencoders
  • ➤ Variational Autoencoders (VAE)
slides(pdf)
Lecture 16
Thu, May 21
Lecture 17
Tue, May 26
Lecture 18
Thu, May 28