Offline Reinforcement Learning: A Comprehensive Guide

Introduction

Machine learning (ML) has revolutionized numerous fields, enabling smarter decisions based on data. However, many real-world applications, such as dynamic pricing in ride-sharing, inventory management in online retail, and autonomous driving, remain challenging for traditional ML techniques. These applications often require manual design of decision rules and state machines. Recent advances in offline reinforcement learning (RL) promise to change this, offering a path to truly data-driven automated decision-making. This article explores the potential of offline RL to bridge the gap between data and action, enabling end-to-end learning in complex, real-world scenarios.

The Limitations of Supervised Learning in Decision Making

Supervised learning systems excel at prediction. For example, a model might forecast a significant increase in customer orders. However, translating these predictions into effective decisions requires human intuition and hand-crafted rules. The model doesn't inherently understand the impact of actions on outcomes.Supervised learning differs from real-world decision-making in the following ways:

Quantity predicted: Supervised learning predicts manually selected quantities, and sequential decision making only specifies the objective manually.
Decision making: Supervised learning requires decisions to be made manually based on predictions, while sequential decision making requires outputting near-optimal actions to achieve desired outcomes.
Data assumptions: Supervised learning assumes independent and identically distributed (i.i.d.) data, whereas sequential decision making acknowledges that each observation is part of a sequential process where actions influence future observations.
Feedback: Supervised learning ignores feedback, but sequential decision making considers feedback critical to achieving desired goals through long-term interaction.

Reinforcement Learning: A Direct Approach to Decision Making

Reinforcement learning (RL) directly addresses the decision-making problem. RL algorithms aim to optimize long-term performance in dynamic environments. They have achieved remarkable success in various domains, from game playing to robotics. However, traditional RL methods typically operate in an active learning setting, where an agent interacts directly with its environment, observes the consequences of its actions, and learns through trial and error.

The classic diagram of reinforcement learning fundamentally represents an active and online learning process. Instantiating this framework with real-world data collection is difficult, because partially trained agents interacting with real physical systems require careful oversight and supervision. For this reason, most of the work that utilizes reinforcement learning relies either on meticulously hand-designed simulators, which preclude handling complex real-world situations, especially ones with unpredictable human participants, or requires carefully designed real-world learning setups, as in the case of real-world robotic learning. More fundamentally, this precludes combining RL algorithms with the most successful formula in ML. From computer vision to NLP to speech recognition, time and time again we’ve seen that large datasets, combined with large models, can enable effective generalization in complex real-world settings. However, with active online RL algorithms that must recollect their dataset each time a new model is trained, such a formula becomes impractical. Here are some of the differences between the active RL setup and data-driven machine learning:

Data collection: Active RL requires the agent to collect data each time it is trained, while data-driven machine learning allows data to be collected once and reused for all models.
Data source: Active RL requires the agent to collect data using its own (partially trained) policy, while data-driven machine learning allows data to be collected with any strategy, including hand-engineered systems, humans, or random exploration.
Dataset size and diversity: Active RL tends to use narrow datasets collected in specific environments or manually designed simulators, while data-driven machine learning can leverage large and diverse datasets collected from all available sources.
Generalization: Active RL can suffer from poor generalization due to small, narrow datasets or simulators that differ from reality, while data-driven machine learning often exhibits good generalization due to large and diverse datasets.

The Challenge of Real-World Implementation

Implementing active RL in the real world poses significant challenges. Partially trained agents interacting with real systems require careful oversight to avoid unintended consequences. Furthermore, creating realistic simulators for complex real-world scenarios, especially those involving human behavior, can be exceedingly difficult.

Offline Reinforcement Learning: Bridging the Gap

Offline reinforcement learning (RL) combines the strengths of reinforcement learning and data-driven machine learning. Offline RL algorithms learn from previously collected data without requiring additional online interaction. This approach addresses the limitations of active RL by enabling the use of large, diverse datasets collected from various sources, such as human demonstrations, existing systems, or random exploration.

The diagram below illustrates the differences between classic online reinforcement learning, off-policy reinforcement learning, and offline reinforcement learning:

In online RL, data is collected each time the policy is modified.
In off-policy RL, old data is retained, and new data is still collected periodically as the policy changes.
In offline RL, the data is collected once, in advance, much like in the supervised learning setting, and is then used to train optimal policies without any additional online data collection.

Crucially, when the need to collect additional data with the latest policy is removed completely, reinforcement learning does not require any capability to interact with the world during training. This removes a wide range of cost, practicality, and safety issues: we no longer need to deploy partially trained and potentially unsafe policies, we no longer need to figure out how to conduct multiple trials in the real world, and we no longer need to build complex simulators. The offline data for this learning process could be collected from a baseline manually designed controller, or even by humans demonstrating a range of behaviors. These behaviors do not need to all be good either, in contrast to imitation learning methods. This approach removes one of the most complex and challenging parts of a real-world reinforcement learning system.

Advantages of Offline RL

Offline RL offers several key advantages:

Utilizing Existing Data: Offline RL can leverage previously collected datasets, including large and diverse datasets that are impractical to collect actively.
Eliminating Online Interaction: By removing the need for online data collection, offline RL avoids the risks and costs associated with deploying partially trained policies in the real world.
Learning from Diverse Behaviors: Offline RL can learn from datasets containing a wide range of behaviors, including suboptimal or exploratory actions, unlike imitation learning methods that require expert demonstrations.

Potential Applications

The ability to learn from offline data opens up a wide range of potential applications:

Autonomous Driving: Training autonomous vehicles on millions of videos of real-world driving.
HVAC Control: Optimizing HVAC systems using logged data from numerous buildings.
Traffic Light Optimization: Controlling traffic lights to improve city traffic flow using data from various intersections.

How Offline Reinforcement Learning Algorithms Work

The fundamental challenge in offline reinforcement learning is distributional shift. The offline training data comes from a fixed distribution (sometimes referred to as the behavior policy). The new policy that we learn from this data induces a different distribution. Every offline RL algorithm must contend with the resulting distributional shift problem. One widely studied approach in the literature is to employ importance sampling, where distributional shift can lead to high variance in the importance weights. Algorithms based on value functions (e.g., deep Q-learning and actor-critic methods) must contend with distributional shift in the inputs to the Q-function: the Q-function is trained under the state-action distribution induced by the behavior policy, but evaluated, for the purpose of policy improvement, under the distribution induced by the latest policy. Using the Q-function to evaluate or improve a learned policy can result in out-of-distribution actions being passed into the Q-function, leading to unpredictable and likely incorrect predictions. When the policy is optimized so as to maximize its predicted Q-values, this leads to a kind of “adversarial example” problem, where the policy learns to produce actions that “fool” the learned Q-function into thinking they are good.

Most successful offline RL methods address this problem with some type of constrained or conservative update, which either avoids excessive distribution shift by limiting how much the learned policy can deviate from the behavior policy, or explicitly regularizes learned value functions or Q-functions so that the Q-values for unlikely actions are kept low, which in turn also limits the distribution shift by dis-incentivizing the policy from taking these unlikely, out-of-distribution actions. The intuition is that we should only allow the policy to take those actions for which the data supports viable predictions.

Addressing Distributional Shift

The primary challenge in offline RL is distributional shift. The training data is generated by a fixed behavior policy, while the learned policy induces a different distribution. This discrepancy can lead to inaccurate value function estimates and poor policy performance.

Several techniques have been developed to mitigate distributional shift:

Policy Constraints: Limiting how much the learned policy can deviate from the behavior policy.
Implicit Constraints: Using weighted maximum likelihood updates to implicitly constrain the policy.
Conservative Q-Functions: Regularizing the Q-function to assign lower values to out-of-distribution actions.

Recent Advances in Offline RL

Building on these ideas, recent advances in offline reinforcement learning have led to substantial improvements in the capabilities of offline RL algorithms. A complete technical discussion of these methods is outside the scope of this article, and I would refer the reader to our recent tutorial paper for more details. However, I will briefly summarize several recent advances that I think are particularly exciting:

Policy Constraints

A simple approach to control distributional shift is to limit how much the learned policy can deviate from the behavior policy. This is especially natural for actor-critic algorithms, where policy constraints can be formalized as using the following type of policy update:

The constraint, expressed in terms of some divergence (“D”), limits how far the learned policy deviates from the behavior policy. Examples include KL-divergence constraints and support constraints. Note that such methods require estimating the behavior policy by using another neural network, which can be a substantial source of error.

Implicit Constraints

The AWR and AWAC algorithms instead perform offline RL by using an implicit constraint. Instead of explicitly learning the behavior policy, these methods solve for the optimal policy via a weighted maximum likelihood update of the following form:

Here, A(s,a) is an estimate of the advantage, which is computed in different ways for different algorithms (AWR uses Monte Carlo estimates, while AWAC uses an off-policy Q-function). Computing the expectation under the behavior policy only requires samples from the behavior policy, which we can obtain directly from the dataset, without actually needing to estimate what the behavior policy is. This makes AWR and AWAC substantially simpler, and enables good performance in practice.

Conservative Q-functions

A very different approach to offline RL, which we explore in our recent conservative Q-learning (CQL) paper, is to not constrain the policy at all, but instead regularize the Q-function to assign lower values to out-of-distribution actions. This prevents the policy from taking these actions, and results in a much simpler algorithm that in practice attains state-of-the-art performance across a wide range of offline RL benchmark problems.

Offline RL vs. Online RL

Deep Reinforcement Learning (RL) is a framework to build decision-making agents. These agents aim to learn optimal behavior (policy) by interacting with the environment through trial and error and receiving rewards as unique feedback. The agent’s goal is to maximize its cumulative reward, called return. Because RL is based on the reward hypothesis: all goals can be described as the maximization of the expected cumulative reward. Deep Reinforcement Learning agents learn with batches of experience. The question is, how do they collect it?:

In online reinforcement learning, the agent gathers data directly: it collects a batch of experience by interacting with the environment. Then, it uses this experience immediately (or via some replay buffer) to learn from it (update its policy). But this implies that either you train your agent directly in the real world or have a simulator. If you don’t have one, you need to build it, which can be very complex (how to reflect the complex reality of the real world in an environment?), expensive, and insecure (if the simulator has flaws that may provide a competitive advantage, the agent will exploit them).

On the other hand, in offline reinforcement learning, the agent only uses data collected from other agents or human demonstrations. It does not interact with the environment. The process is as follows:

Create a dataset using one or more policies and/or human interactions.
Run offline RL on this dataset to learn a policy

This method has one drawback: the counterfactual queries problem. What do we do if our agent decides to do something for which we don’t have the data? For instance, turning right on an intersection but we don’t have this trajectory.

tags: #offline #reinforcement #learning #tutorial