Wen Sun

I'm an Assistant Professor in the Computer Science Department at Cornell University. I'm also a part-time researcher at Microsoft Research NYC.

Prior to Cornell, I was a post-doc researcher at Microsoft Research NYC from 2019 to 2020. I completed my PhD at Robotics Institute, Carnegie Mellon University in June 2019, where I was advised by Drew Bagnell.

CV  /  PhD Thesis  /  Google Scholar  /  Email  

My group works on Reinforcement Learning, AI, and Decision Making. The most recent research directions of the lab are

Fall 2024: CS 6789 Foundations of Reinforcement Learning

Fall 2023: CS 4780/5780 Introduction to Machine Learning

Spring 2021: CS 4789/5789 Introduction to Reinforcement Learning

We are periodically making updates to the book draft. Content based on the courses taught by Nan at UIUC, the courses taught by Alekh and Sham at UW, and CS 6789 at Cornell.

Just as GANs led to GAIL in IRL, how could diffusion models --- a more powerful generative model, be applied to IRL to achieve similar success? We provide an answer in this work.

We provide a new RL policy optimization algorithm for multi-turn RLHF. The algorithm enables efficient optimization for a 8B size model in long conversations against a 70B model.

New offline RLHF algorithms that avoid overoptimization and achieve single policy coverage style guarantees by regularizing with the Chi-squared divergence.

We show that resetting to offline data is an effective way of leveraging offline data in the RLHF pipeline

A new framework -- RL with Guided Feedback (RLGF), combining RL and pre-trained LLMs via principled interactive learning procedures.

New RL algorithm for generative model optimization which outperforms PPO on both text and image generation. State-of-art performance on LLM benchmarks.

We study why DPO is not equivalent to RLHF, and what is the key benefit of online samples in RLHF. The new hybrid algorithm that uses both online and offline dadta outperforms DPO on standard RLHF benchmarks.

We show that distributional RL enables faster learning when the systems have low variance. This holds for contextual bandits, online and offline RL simoutaneously.

A light weight, real-world database query optimization benchmark for RL

A randomized algorithm for learning from preference feedback that achieves sample, computation, and query efficiency simoutaneously.

We provide the first mathematical and rigorous explaination of why and when maximum-likelihood-estimation based distributional RL can be better than regular RL, in contextual bandits, online RL, and offline RL. The new distributional contextual bandit algorithm outperforms prior CB algorithms empirically.

A simple maximum likelihood estimation based approach for distributional RL in off-policy policy evaluation with finite sample complexity guarantees

Combining online data and offline data can solve RL with both statistical and computation efficiency. Experiments on Montezuma's Revenge (a video game) reveals that hybrid RL works much better than pure online RL and pure offline RL

A model-based RL algorithm that solves PSRs (a model that generalizes POMDPs) with polynomial sample complexity with general function approximation.

A general model-free Actor-critic framework for POMDPs which generalizes special instances including tabular POMDPs, Linear Quadratic Gaussians, POMDPs with Hilbert Space Embeddings, and POMDPs with low-rank structures.

Standard self-supervised representation learning approaches fail to work in offline RL due to distribution shift and the sequential nature of the problem. Our new self-supervised representation learning approach works in theory and in practice for offline RL.

An efficient rich-observation RL algorithm that learns to decode from rich observations to latent states (via adversarial training), while balancing exploration and exploitation

Investigated when ILC learns a policy that is better than the certainty equivanece policy

Interleaving representation learning, exploration, and exploitation for efficient RL with nonlinear function approximation

We show partial coverage and realizability is enough for efficient model-based learning in offline RL; notable examples include low-rank MDPs, KNRs, and factored MDPs.

We show how to mitigate covariate shift by leveraging offline data that only provides partial coverage. A by-product of this work is new results for offline RL: partial coverage and robustness (i.e., being able to compete agaist any policy covered by the offline data)

IL from Observations is strictly harder than the classic IL; we incoporate exploration into the min-max IL framework (we balance exploration and imitation) to solve IL from observations near optimally in theory and efficiently in practice.

A general framework that enables (1) active action elimination in RL, and (2) enables provably robust exploration with adversarial corruptions on both rewards and transitions.

An On-policy algorithm that is robust to constant fraction adversarial corruption; The TRPO/NPG based implementation scales to high-dimension control tasks and is robust to strong data corruption.

We propose a simple model-based algorithm that achieves state-of-art in both dense reward continuous control tasks and sparse reward control tasks that require efficient exploration

A new structural complexity captures generalization in RL with function approximation in both model-free and model-based settings. Notably, we show that MDPs with linear Q* and linear V* is PAC learnable.

We study the advantages of on-policy policy gradient methods compared to off-policy methods such as Q-learning, and provide a new PG algorithm with exploration

We study learning-to-control for nonlinear systems captured by RKHS or Gaussian Processes. While being more general, the regret bound is near optimal when specialized to LQRs

Representation Learning in RL needs to be done jointly with exploration; we show how to do this correctly and riguously

We study multi-objective RL and show how to do cautious exploration under various constraints

We study Sim-to-Real under a model-based framework resulting an algorithm that enjoyes strong theoretical guarantees and excellent empirical performance

We unify Imitation Learning by casting it as f-divergence minimization problem

Using disagreement among an ensemble of pre-trained behavior cloning policies to reduce covariate shift in IL

Frame IL with observations alone as a sequence of two-player minmax games.
Polynomial sample complexity for learning near-optimal policy with general function approximation.

An incremental & learnable memory system maintained in a nearly balanced tree structure to ensure logarithmic time operations

A theoretical comparison between model-based RL and model-free RL.
A sample efficient model-based RL algorithm.

Exploration in action space can be much more efficient than zero-th order method when the number of policy parameters is way larger than the dimension of action space and planning horizon.

Leverage Model-based control (i.e., iLQR) and reward-aware Imitation Learning (e.g., AggreVaTeD) to double boost policy improvement

Combination of IL & RL: use expert's value function as reward shaping to shorten planning horizon which in turn speeds up RL

Can be viewed as an actor-critic algorithm with critic being expert's state-action Q function; exponential sample complexity seperation between IL and pure RL

Minizing Regret while maintaining average risk below a pre-specified safety-threshold in long term

Learning to Filter with Predictive State Inference Machines
Wen Sun, Arun Venkatraman, Byron Boots, J. Andrew Bagnell.
ICML 2016 [slides]

Learning to predict future recurrently.
Can be viewed as a recurrent structure whose hidden state encodes information for accurately predicting future

Online Bellman Residual Algorithms with Predictive Error Guarantees
Wen Sun, Drew Bagnell
UAI, 2015   Best Student Paper Award [slides]

Adversarial online policy evaluation.
A reduction from adversarial policy evaluation to general no-regret & stable online learning.

Template from here