Rui Liu Wins CISE 2022 Best Student Paper Award
Rui Liu, 5th year Ph.D. Candidate in 系统工程, won the 2022 CISE Best Paper Award for her paper titled Temporal Difference Learning as Gradient Splitting. Liu’s interests consist of reinforcement learning, multi-agent systems, and optimization.
Her paper gives a fuller explanation for why a common class of algorithms in reinforcement learning work as well as they do. Reinforcement learning is a type of machine learning that has been applied to autonomous driving, robotics, bidding and advertising, and games. The goal of reinforcement learning is to find an optimal policy in a situation where actions taken by an agent affect the agent’s state and ability to take future actions . This is where temporal difference learning comes in. Algorithms using temporal difference learning as a subroutine are widely used in reinforcement learning.
“Rui’s research has shed new light on a canonical algorithm in reinforcement learning. We are still thinking through all of its implications, but it is likely to have repercussions for many other methods in reinforcement learning, and could lead to the development of entirely new algorithms,” advisor Alex Olshevsky said.
The key new idea in the paper is to interpret temporal difference learning through the lens of a new concept called a “gradient splitting,” introduced in the paper. This leads to a clearer and sharper analysis of temporal difference methods.
“This work is theoretical. So it explains why this algorithm works and it gives you a better convergence time. When doing work, the proof results may not be as good as this algorithm is in the practice, but our analysis improves the current theoretical results,” Liu said.
Liu hopes her work will improve the state of the art in many robotics applications where temporal difference learning is used.
Liu got her Masters from the Chinese Academy of Sciences and said reinforcement learning interests her because it has many real world applications.