Events & News

🌟 March 2025 🌟 - New particularly exciting preprint out on the real value of RL in fine-tuning / RLHF. I gave a talk at Cornell on the paper that might also be of interest.

January 2025 - Three papers accepted to ICLR’25! One on using score matching for inverse RL, one on REFUEL: a deceptively simple multi-turn RLHF algorithm, and one on what the right reset distribution is for speeding up inverse RL.

🌟 November 2024 🌟 - Drew, Steven, and I are co-teaching a course on the algorithmic foundations of interactive learning. If you’d like to understand the fundamental principles beyond imitation (e.g. for robots) and RLHF (e.g. for LLMs), this is the course for you!

September 2024 - Three papers accepted to NeurIPS’24! One on what is fundamentally different about multi-agent imitation learning, one on REBEL: a scalable and theoretically elegant RLHF algorithm, and one on the differences between online and offline preference fine-tuning algorithms.

June 2024 - Four papers accepted to ICML’24! One on SPO: an RLHF algorithm for reconciling diverse preferences, one on a fundamental framework for designing efficient inverse RL algorithms, one on evolving reward shaping terms for RL / Inverse RL, and one on transfer learning.


Research Highlights

All Roads Lead to Likelihood

We explore how the value of RL in fine-tuning / RLHF seems to be fundamentally derived from generation-verification gaps. [Paper] [Talk]

SPO: Self-Play Preference Optimization

We derive a new fundamental algorithm for RLHF that robustly handles the complex, intransitive preferences that often result from aggregating the a diversity of views. [Website] [Paper]

Inverse RL without RL

We derive exponentially faster algorithms for inverse RL by proving that local search, rather than global RL, is "all you need" for imitation. [Website][Paper]

Imitating a Privileged Expert

We formalize what the value of interaction (e.g DAgger) is when imitating an expert with access to privileged information (e.g. simulator state) using techniques from causal inference. [Website][Blog]