Niao He - Reinforcement Learning: Optimization and Dynamical Systems Perspectives

Reinforcement learning (RL) has been in the limelight with many recent breakthroughs in artificial intelligence, yet theoretical understanding of many classical RL algorithms and their modern deep RL counterparts remain quite limited. Recently, fundamental concepts from optimization and dynamical systems theory have provided fresh perspectives to understanding their convergence behaviors as well as developing provably efficient RL algorithms.

This short course will introduce some recent theoretical and algorithmic developments in RL from the optimization and dynamical systems perspectives. Tentative topics include

  • Overview of RL basics (dynamic programming, TD-learning, Q-learning, policy gradient, etc.)
  • Unified O.D.E. analysis of RL algorithms (TD-learning and Q-learning variants)
  • Stochastic optimization frameworks for RL
  • Primal-dual optimization frameworks for RL
  • Optimization for RL from a single agent to cooperative agents
  • Extension to the theory of deep RL
  • Open problems

Papers for the workshops (click to access):

Click here to access the slides
Yurii Nesterov - Modern Theory of Second-Order Methods

In this course we present the latest achievements in the theory of high-order schemes. We start from the global complexity estimates for the second-order methods, based on then cubic regularization of the quadratic model of the objective function. We consider several complexity bounds for different classes of nonconvex functions. Next topic is the accelerated second-order methods for convex functions and the lower complexity bounds. After that, we pass to the universal second order schemes. And we finish the course with implementable tensor methods.

Papers for the workshops (click to access):

Lecture material
Schedule