Building Up RL
From Dynamics and Control to Learning
The book for my graduate course IFT6162. It starts from dynamics, trajectory optimization and model predictive control, and builds up to reinforcement learning.
Pierre-Luc Bacon
Associate Professor, Université de Montréal and Mila
I'm an Associate Professor at the Université de Montréal's DIRO, a core member of Mila, and I hold a Canada CIFAR AI Chair. I'm also affiliated with IVADO. I did my PhD at McGill with Doina Precup and a postdoc at Stanford with Emma Brunskill.
My research is in reinforcement learning and optimal control, with a long-standing interest in the curse of the horizon and how representation learning can help overcome it. Lately I've been especially drawn to applications: using RL to improve HVAC and energy systems, to discover new materials, and to accelerate drug discovery. On the methodological side, I try to meld ideas from optimal control and RL; my course book, Building Up RL, explains the motivation.
Course books
Fondements de l'apprentissage machine
Un manuel d'introduction à l'apprentissage machine
The book for IFT3395/IFT6390: supervised learning and generalization, the probabilistic view, neural networks and ensemble methods.
Publications
77 papers
2026
-
From Static Policies to Adaptive Priors in Offline Reinforcement Learning
NeurIPS 2026, Position Paper Track
-
Representation Learning Enables Scalable Multitask Deep Reinforcement Learning
NeurIPS 2026
-
Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priors
NeurIPS 2026, Position Paper Track
-
Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity
ICML 2026Spotlight
-
What Makes Value Learning Efficient in Residual Reinforcement Learning?
ICML 2026Spotlight
-
Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism
ICML 2026code
-
Towards Practical World Model-based Reinforcement Learning for Vision-Language-Action Models
ICML 2026
-
Discrete Compositional Generation via General Soft Operators and Robust Reinforcement Learning
ICLR 2026
-
Building2Building: A Large Scale Benchmark for Generalizable Real-World Reinforcement Learning
RLC 2026
-
Rotation-Preserving Supervised Fine-Tuning
ICML 2026 Workshop on Continual Adaptation at Scale
-
Layerwise LQR for Geometry-Aware Optimization of Deep Networks
Preprint
-
Do Tabular Foundation Models Know Physics? Contamination, Units, and the Deterministic Limit
Preprint
2025
-
State Entropy Regularization for Robust Reinforcement Learning
NeurIPS 2025Oral
-
Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning
NeurIPS 2025Spotlight
-
Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning
ICML 2025Oral
-
Scaling Trends in Language Model Robustness
ICML 2025Spotlight
-
MaestroMotif: Skill Design from Artificial Intelligence Feedback
ICLR 2025Oral
-
Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons
Transactions on Machine Learning Research (TMLR), 2025arXiv
-
Understanding Behavioral Metric Learning: A Large-Scale Study on Distracting Reinforcement Learning Environments
RLC 2025
-
The Three Regimes of Offline-to-Online Reinforcement Learning
Preprint
-
Planning with Unified Multimodal Models
Preprint
-
Discovery of Sustainable Refrigerants through Physics-Informed RL Fine-Tuning of Sequence Models
EurIPS 2025 Workshop on Simulation for Biology and Chemistry (SIMBIOCHEM)code
-
Mol-MoE: Training Preference-Guided Routers for Molecule Generation
ICLR 2025 Workshop on Generative and Experimental Perspectives for Biomolecular Design (GEM)code
2024
-
Do Transformer World Models Give Better Policy Gradients?
ICML 2024
-
Bridging State and History Representations: Understanding Self-Predictive RL
ICLR 2024
-
Decoupling Regularization from the Action Space
ICLR 2024
-
Course Correcting Koopman Representations
ICLR 2024
-
Motif: Intrinsic Motivation from Artificial Intelligence Feedback
ICLR 2024
-
Maximum Entropy GFlowNets with Soft Q-learning
AISTATS 2024
-
Neural Differential Equations for Temperature Control in Buildings under Demand Response Programs
Applied Energy, vol. 368, 2024
-
Generative Active Learning for the Search of Small-molecule Protein Binders
Preprint
2023
-
When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment
NeurIPS 2023Oral
-
Double Gumbel Q-Learning
NeurIPS 2023Spotlight
-
Block-State Transformers
NeurIPS 2023
-
Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control
NeurIPS 2023
-
Sample-Efficient Reinforcement Learning by Breaking the Replay Ratio Barrier
ICLR 2023Notable top 5%
2022
-
Myriad: A Real-World Testbed to Bridge Trajectory Optimization and Deep Learning
NeurIPS 2022, Datasets and Benchmarks Track
-
The Primacy Bias in Deep Reinforcement Learning
ICML 2022 and RLDM 2022
-
Direct Behavior Specification via Constrained Reinforcement Learning
ICML 2022
-
Continuous-Time Meta-Learning with Forward Mode Differentiation
ICLR 2022
-
Control-Oriented Model-Based Reinforcement Learning with Implicit Differentiation
AAAI 2022
2021
-
Neural Algorithmic Reasoners are Implicit Planners
NeurIPS 2021
-
Meta Dynamic Programming
NeurIPS 2021 Workshop on Metacognition in the Age of AI
-
Long-Term Credit Assignment via Model-based Temporal Shortcuts
NeurIPS 2021 Deep Reinforcement Learning Workshop
2020
-
Understanding the Curse of Horizon in Off-Policy Evaluation via Conditional Importance Sampling
ICML 2020
-
Options of Interest: Temporal Abstraction with Interest Functions
AAAI 2020
-
Counterfactual Policy Evaluation and the Conditional Monte Carlo Method
NeurIPS 2020 Offline Reinforcement Learning Workshop
-
TDprop: Does Jacobi Preconditioning Help Temporal Difference Learning?
ICML 2020 Workshop on Theoretical Foundations of Reinforcement Learning
-
Policy Evaluation Networks
Preprint
2019
-
A Lagrangian Method for Inverse Problems in Reinforcement Learning
NeurIPS 2019 Workshop on Optimization Foundations for Reinforcement Learning
-
All-Action Policy Gradient Methods: A Numerical Integration Approach
NeurIPS 2019 Workshop on Optimization Foundations for Reinforcement Learning
-
Goal-Directed Learning as a Bi-level Optimization Problem
RLDM 2019
2018
-
Temporal Representation Learning
PhD thesis, McGill University, 2018
-
Convergent Tree-Backup and Retrace with Function Approximation
ICML 2018arXiv
-
When Waiting is not an Option: Learning Options with a Deliberation Cost
AAAI 2018arXiv
-
Learning with Options that Terminate Off-Policy
AAAI 2018arXiv
-
OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learning
AAAI 2018arXiv
-
Learning Robust Options
AAAI 2018
-
Constructing Temporal Abstractions Autonomously in Reinforcement Learning
AI Magazine, vol. 39, no. 1, 2018
2017
- The Option-Critic Architecture
-
Unifying Multi-Step Methods through Matrix Splitting
RLDM 2017
-
Learning Options End-to-End for Continuous Action Tasks
NIPS 2017 Hierarchical Reinforcement Learning Workshop
-
A Unified View on Multi-Step Methods using Matrix Splittings
Data Learning and Inference (DALI), 2017
2016
-
A Matrix Splitting Perspective on Planning with Options
NIPS 2016 Continual Learning and Deep Networks Workshopposter
-
Advances in Option Construction: The Option-Critic Architecture
ICML 2016 Abstraction in Reinforcement Learning Workshopvideo
-
The Good, the Bad and the Discovery: The Specification Problem of Options Discovery
10th Barbados Workshop on Reinforcement Learning, 2016
-
Conditional Computation in Neural Networks for Faster Models
Preprint
2015
-
Learning and Planning with Timing Information in Markov Decision Processes
UAI 2015poster
-
Learning with Options: Just Deliberate and Relax
NIPS 2015 Workshop on Bounded Optimality and Rational Metareasoningposter
-
The Option-Critic Architecture
NIPS 2015 Deep Reinforcement Learning Workshopposter
-
Analyzing Open Data from the City of Montreal
ICML 2015 Workshop on Mining Urban Data
-
Conditional Computation in Neural Networks using a Decision-Theoretic Approach
RLDM 2015
-
Learning and Planning with Timing Information in Markov Decision Processes
RLDM 2015
-
Learning Recognizers
9th Barbados Workshop on Reinforcement Learning, 2015slides
2014
-
Predictive Timing Models
NIPS 2014 Workshop From Bad Models to Good Policiesposterslidesvideo
2013
-
Using Label Propagation for Learning Temporally Abstract Actions in Reinforcement Learning
AAMAS 2013 Workshop on Multiagent Interaction Networks
-
On the Bottleneck Concept for Options Discovery: Theoretical Underpinnings and Extension in Continuous State Spaces
MSc thesis, McGill University, 2013
No papers match this filter. Try an author's last name, a venue such as ICML, or a year.