MB-DQN uses different backup lengths for improved reinforcement learning.
problem Improving reinforcement learning with multi-step returns.
method Integrates multi-step returns into bootstrapped DQN with different backup lengths.
result MB-DQN maintains advantages of different backup lengths.
Multi-step methods such as Retrace(λ) and n-step Q-learning have become a crucial component of modern deep reinforcement learning agents. These methods are often evaluated as a part of bigger architectures and their evaluations rarely include enough samples to draw statistically significant conclusions about thei…
A new algorithm reduces training time for distributed machine learning by dynamically assigning backup workers.
problem Time-consuming synchronization phase due to slow workers (stragglers).
method Dynamic allocation of backup workers to minimize waiting time.
result Achieves linear speedup in convergence performance with more workers.
Decouples critic chunk length from policy to improve policy reactivity and performance.
problem Bootstrapping bias and difficulty in extracting optimal policies from chunked critics.
method Optimizes policy against a distilled critic for partial action chunks, allowing shorter chunks for policy.
result Reliably outperforms prior methods on long-horizon offline goal-conditioned tasks.
The study proposes using TD error for selecting σ in Q(σ, λ).
problem Selecting the value of σ in Q(σ, λ) based on state characteristics.
method TD error as a heuristic for selecting σ.
result TD error can effectively select σ based on state characteristics.
The project intends to model the stability of power system with a deep learning algorithm to the problem, aiming to delay the removal of the fault. The so-called "fail-delay cut-off" refers to the occurrence of N-1 backup protection action on the backbone network of the system, resulting in longer time for the removal …
A reinforcement learning framework combining value function and tree search planner for strategic and tactical decisions.
problem Strategic and tactical decision-making in discrete environments.
method Combines value function and tree search planner, using uncertainty modeling and risk measurement.
result Improves performance and learning speed on hard exploration environments.
Modeling supply chain disruptions from climate hazards with adaptive firms.
problem Systemic physical climate risk in supply chains.
method Agent-based model integrating geospatial hazards and firm adaptation.
result Firms' adaptive strategies reduce disruption by 48%.
New RL approach infers optimal policies via variational inference.
problem Manual design of reward functions for reinforcement learning.
method Variational inference for inferring policies achieving desired outcomes.
result Eliminates need for hand-crafted reward functions for diverse tasks.
Modern reinforcement learning algorithms reach super-human performance on many board and video games, but they are sample inefficient, i.e. they typically require significantly more playing experience than humans to reach an equal performance level. To improve sample efficiency, an agent may build a model of the enviro…
Off-policy reinforcement learning aims to leverage experience collected from prior policies for sample-efficient learning. However, in practice, commonly used off-policy approximate dynamic programming methods based on Q-learning and actor-critic methods are highly sensitive to the data distribution, and can make only …
Reinforcement learning has attracted great attention recently, especially policy gradient algorithms, which have been demonstrated on challenging decision making and control tasks. In this paper, we propose an active multi-step TD algorithm with adaptive stepsizes to learn actor and critic. Specifically, our model cons…
Recently, a new multi-step temporal learning algorithm, called Q(σ), unifies n-step Tree-Backup (when σ=0) and n-step Sarsa (when σ=1) by introducing a sampling parameter σ. However, similar to other multi-step temporal-difference learning algorithms, Q(σ) needs much memory consumption and computation tim…
MLaaS (ML-as-a-Service) offerings by cloud computing platforms are becoming increasingly popular. Hosting pre-trained machine learning models in the cloud enables elastic scalability as the demand grows. But providing low latency and reducing the latency variance is a key requirement. Variance is harder to control in a…
Combining deep model-free reinforcement learning with on-line planning is a promising approach to building on the successes of deep RL. On-line planning with look-ahead trees has proven successful in environments where transition models are known a priori. However, in complex environments where transition models need t…
Paper studies offline RL with linear approx, focusing on inherent Bellman error.
problem Offline RL with linear approx, focusing on inherent Bellman error.
method Algorithm that succeeds under single-policy coverage condition, leveraging inherent Bellman error.
result Algorithm yields first known guarantee under single-policy coverage, even for linear Bellman completeness.
Value-based reinforcement learning (RL) methods like Q-learning have shown success in a variety of domains. One challenge in applying Q-learning to continuous-action RL problems, however, is the continuous action maximization (max-Q) required for optimal Bellman backup. In this work, we develop CAQL, a (class of) algor…
The paper analyzes off-policy TD-learning using generalized Bellman operators and provides finite-sample bounds.
problem High variance in off-policy TD-learning due to importance sampling.
method Derives finite-sample bounds for off-policy TD-like algorithms using generalized Bellman operators.
result First-known finite-sample guarantees for several off-policy TD algorithms.
Reducing the latency variance in machine learning inference is a key requirement in many applications. Variance is harder to control in a cloud deployment in the presence of stragglers. In spite of this challenge, inference is increasingly being done in the cloud, due to the advent of affordable machine learning as a s…
Off-policy reinforcement learning with eligibility traces is challenging because of the discrepancy between target policy and behavior policy. One common approach is to measure the difference between two policies in a probabilistic way, such as importance sampling and tree-backup. However, existing off-policy learning …
Reinforcement learning methods carry a well known bias-variance trade-off in n-step algorithms for optimal control. Unfortunately, this has rarely been addressed in current research. This trade-off principle holds independent of the choice of the algorithm, such as n-step SARSA, n-step Expected SARSA or n-step Tree bac…
SUNRISE improves off-policy RL algorithms by integrating ensemble methods.
problem Stability and exploration issues in off-policy RL algorithms.
method SUNRISE combines ensemble-based weighted Bellman backups and upper-confidence bounds for efficient exploration.
result SUNRISE improves the performance of off-policy RL algorithms across various domains.
QAM uses adjoint matching to optimize continuous-action RL policies efficiently.
problem Efficient optimization of expressive diffusion or flow-matching policies with respect to a Q-function.
method QAM leverages adjoint matching to bypass the numerical instability of backpropagation through multi-step denoising processes.
result QAM consistently outperforms prior approaches on hard, sparse reward tasks in offline and offline-to-online RL.
Reinforcement learning is a promising approach to synthesizing policies for challenging robotics tasks. A key problem is how to ensure safety of the learned policy---e.g., that a walking robot does not fall over or that an autonomous car does not run into an obstacle. We focus on the setting where the dynamics are know…
Simplifies BCQ to match and outperform state-of-the-art in offline RL benchmarks.
problem Sample efficiency in offline reinforcement learning.
method Introduces EMaQ, a novel backup operator for offline RL.
result EMaQ matches and outperforms prior state-of-the-art in offline RL benchmarks.
FOCAL tackles offline meta-reinforcement learning with efficient task inference and behavior regularization.
problem Efficiently adapt RL algorithms to unseen tasks without interactions, addressing bootstrapping errors and robust task inference.
method FOCAL combines behavior regularization, a deterministic context encoder, and a negative-power distance metric for efficient task inference.
result FOCAL outperforms prior algorithms on meta-RL benchmarks, demonstrating computational efficiency.
RiskNet predicts penalties in unreliable communication networks using GNNs.
problem Predicting penalties in networks with unreliable resources.
method Graph Neural Network (GNN) based approach trained on random graphs.
result Precisely models penalties across various network topologies.
Paper proposes a second-order method for faster SVI convergence.
problem Poor convergence rate of first-order SVI algorithms.
method Derives Hessian matrix and implements two numerical schemes for efficient second-order SVI.
result Proposed approach achieves faster convergence compared to first-order SVI.
Improved risk-sensitive RL with exponential Bellman equation and better regret bounds.
problem Exponential gap between upper and lower bounds in risk-sensitive RL.
method Identified and addressed deficiencies in existing algorithms and analysis; developed novel analysis and exploration mechanism.
result Improved regret upper bounds over existing ones.
Many currently deployed Reinforcement Learning agents work in an environment shared with humans, be them co-workers, users or clients. It is desirable that these agents adjust to people's preferences, learn faster thanks to their help, and act safely around them. We argue that most current approaches that learn from hu…
Safe autonomous decisions made with machine learning predictions using Conformal Decision Theory.
problem Safe decisions from imperfect machine learning predictions.
method Conformal Decision Theory framework for producing safe decisions.
result Safe decisions with provable statistical guarantees of low risk.
Knoop enhances variable selection with over-parameterization and knockoffs.
problem Challenges of variable selection in high-dimensional datasets.
method Generates knockoff variables, integrates them into an over-parameterized model, and uses anomaly-based significance tests.
result Superior performance in variable selection compared to existing methods.
Adversarial training is a principled approach for training robust neural networks. Despite of tremendous successes in practice, its theoretical properties still remain largely unexplored. In this paper, we provide new theoretical insights of gradient descent based adversarial training by studying its computational prop…
Q-chunking improves RL for long tasks by chunking actions.
problem Improving sample efficiency and exploration in offline-to-online RL.
method Action chunking in TD-based RL methods.
result Q-chunking outperforms prior methods on long-horizon tasks.
Gradient descent biases towards stable rank networks for nearly-orthogonal data.
problem Understanding implicit bias in non-smooth neural networks trained by gradient descent.
method Analysis of two-layer ReLU and leaky ReLU networks trained by gradient descent on nearly-orthogonal data.
result Gradient descent biases towards networks with stable rank and uniform margin for nearly-orthogonal data.
TT-DAC-PS: A deterministic actor-critic approach for optimal trade execution
problem Optimal execution of large stock sell programs
method Twin-Target Deterministic Actor-Critic with Policy Smoothing
result Reduces mean implementation shortfall percentage
Hydropower reduces system electricity price and volatility, especially at extreme levels.
problem Impact of hydropower on system electricity price and volatility.
method Robust statistical analysis using multiple linear regression and quantile regression.
result Hydropower reduces system electricity price and volatility, especially at extreme levels.
Work addresses long-term accuracy issues in IoT air quality sensors.
problem Limited accuracy of IoT air quality sensors in long-term field deployments.
method Adaptive machine learning strategies for network calibration.
result Prolongs the validity of multisensor calibration models for continuous learning.
Machine learning (ML) techniques are increasingly applied to decision-making and control problems in Cyber-Physical Systems among which many are safety-critical, e.g., chemical plants, robotics, autonomous vehicles. Despite the significant benefits brought by ML techniques, they also raise additional safety issues beca…
Paper introduces a method to evaluate abstaining classifiers by considering missing predictions as counterfactuals.
problem Lack of a principled approach to evaluate and compare abstaining classifiers, especially when missing predictions are important.
method Develops a novel approach to treat abstentions as missing data, defining counterfactual scores and using causal inference methods to estimate them.
result Shows that under certain conditions, counterfactual scores can be identified and estimated efficiently, improving evaluation of abstaining classifiers.
New algorithm learns sparse linear MDPs with polynomial interactions, improving sample complexity.
problem Learning optimal policies in sparse linear MDPs with limited interactions and unknown features.
method Developed a polynomial-time algorithm using feature selection and emulator for sparse linear MDPs.
result First polynomial-time algorithm for learning near-optimal policies in k-sparse linear MDPs.
New method calculates Shapley values for uncertain functions.
problem Uncertain value functions in explainable machine learning.
method Definition of Shapley values using probability theory.
result Shapley values can be applied to uncertain functions.
New set-valued star-shaped risk measures introduced for better risk assessment.
problem Improving risk assessment in financial contexts.
method Developed new set-valued star-shaped risk measures and proved their representation theorems.
result Set-valued star-shaped risk measures can be represented as unions of set-valued convex risk measures.
The paper introduces Absolute Shapley Value to handle negative contributions in machine learning model training.
problem Negative marginal contributions in machine learning model training.
method Investigates three philosophies: Original Shapley Value, Zero Shapley Value, and Absolute Shapley Value.
result Absolute Shapley Value significantly outperforms other definitions in evaluating data importance.
Study on 2-valued dynamics on complex plane, showing some dynamics can't be group actions.
problem Whether 2-valued dynamics can be defined by the action of a 2-valued group.
method Construction of examples of dynamics that are or are not group actions.
result Some 2-valued dynamics on complex plane cannot be defined by the action of a 2-valued group.
Formula for Z_2-valued index of symmetric operators on manifolds.
problem Index of symmetric operators on manifolds with boundary.
method Cohomological formula for Z_2-valued index.
result A formula for the Z_2-valued index of operators on manifolds.
New method converts p-values to e-values for more efficient CP and aggregation.
problem Limitations of existing p-to-e calibrators in CP setting.
method Proposes a novel P2E calibrator for set-preserving calibration.
result Significant efficiency gains over existing p-to-e calibrators.
Introduces joint Shapley values to measure feature importance in models.
problem Measuring the importance of feature sets in machine learning models.
method Extends Shapley's axioms to measure a set of features' average contribution to a model's prediction.
result Joint Shapley values provide unique insights and are more consistent with local intuitions.