Data poisoning attacks can manipulate contextual bandit algorithms to pull target arms.
problem Offline data poisoning attacks in contextual bandits.
method General attack framework based on convex optimization.
result Attacker can force bandit algorithm to pull target arm for target contextual vector.
Efficiently poisons offline RLHF models by flipping preference labels.
problem Vulnerability of offline RLHF models to preference label flipping attacks.
method Developed two attack methods: BAL-A and BMP-A, solving a structured binary sparse approximation problem.
result Demonstrated that flipping one preference label induces a parameter-independent shift in the DPO gradient, enabling structured binary sparse approximation.
RORL improves offline RL robustness with conservative smoothing.
problem Distribution shift and robustness issues in offline RL.
method RORL introduces regularization and conservative smoothing for robustness.
result RORL achieves state-of-the-art performance and robustness to adversarial perturbations.
Adversaries can poison online learning data to mislead models.
problem Adversarial attacks on online learning systems.
method Formalized problem into two settings, proposed optimization-based attack strategy, and three solution strategies.
result Demonstrated effectiveness of data poisoning attacks in online learning.
Paper studies attacks on bandit algorithms and shows how attackers can manipulate data to hijack behavior.
problem Potential attacks on bandit algorithms can cause catastrophic loss in real-world applications.
method Proposes a framework of offline and online attacks on bandit algorithms using convex optimization and adaptive strategies.
result Attackers can force bandit algorithms to pull target arms with high probability by manipulating data.
Attackers can poison environments to force RL agents to follow target policies.
problem Security threat to reinforcement learning where attackers manipulate environments to force agents into following target policies.
method Optimization framework for finding optimal stealthy attacks under different measures of attack cost.
result Attackers can easily succeed in teaching any target policy to RL agents under mild conditions.
Optimal policies identified for learning systems with a malicious expert.
problem Adversarial attacks on learning systems combining expert advice.
method Analysis of offline and online settings, dynamic programming for online setting.
result Greedy policy is asymptotically optimal with approximation ratio for offline setting.
MLCapsule protects sensitive data by executing models locally.
problem Protecting sensitive data and model security in MLaaS.
method Local execution of models on client side, coupled with security measures.
result Models executed locally without data leaving the client, protecting against attacks.
Recent studies have shown that adversarial examples in state-of-the-art image classifiers trained by deep neural networks (DNN) can be easily generated when the target model is transparent to an attacker, known as the white-box setting. However, when attacking a deployed machine learning service, one can only acquire t…
Novel asynchronous SGD method resists Byzantine attacks without server storage.
problem Asynchronous distributed learning with Byzantine attacks and failures.
method Buffered Asynchronous SGD (BASGD) and its momentum variant (BASGDm).
result BASGD and BASGDm resist non-omniscient and omniscient attacks without server storage.
Study fusion methods for financial image views to improve robustness against attacks.
problem Improving robustness of financial image views for next-day direction prediction.
method Same-source multi-view learning with early fusion and late fusion, using OHLCV and technical-indicator views, and evaluating pixel-space L-infinity attacks.
result Early fusion can suffer negative transfer under noisy settings, while late fusion is more reliable once labels stabilize.
New NIDS uses hypergraphs for real-time detection of evolving port scans.
problem Challenges in offline NIDS for auto-generated port scans.
method Hypergraph-based metrics for real-time ML ensemble NIDS.
result High accuracy, precision, and recall in detecting port scans and attacks.
SARD improves adversarial robustness in two-stage L2D systems.
problem Adversarial attacks can manipulate query allocation in two-stage L2D systems.
method Introduces SARD, a convex learning algorithm with provable guarantees.
result SARD significantly improves robustness under adversarial attacks while maintaining strong clean performance.
Paper tackles offline meta-reinforcement learning with a new algorithm.
problem Performing reinforcement learning on limited data from a new task.
method Meta-Actor Critic with Advantage Weighting (MACAW) algorithm.
result Achieves notable gains over prior methods on offline meta-RL benchmarks.
Algorithm balances online and offline data for linear bandits.
problem Online learning with an offline dataset in linear bandits.
method Proposes a linear bandit algorithm that uses offline data early and increasingly favors exploration as the horizon grows.
result Establishes regret bounds showing competitive performance with both purely online and offline solutions.
Algorithm learns from offline data to improve performance in target environment.
problem Learning from offline data in a target environment with unknown shifts.
method Adaptive algorithm that uses offline data to improve performance when informative.
result Algorithm provably improves performance over purely online learning when offline data are informative.
Algorithm reduces online regret by leveraging offline data in linear bandits.
problem Online regret minimization in linear bandits with offline data.
method OOPE algorithm using extended D-optimal design.
result Substantial reduction in online regret compared to prior work.
New method selects best offline RL policies from logged data.
problem Hyperparameter selection challenges offline RL.
method Offline hyperparameter selection for RL algorithms.
result Reliable ranking and selection of policies across hyperparameters.
Optimized RL algorithms perform well on offline datasets, outperforming fully trained agents.
problem Improving reinforcement learning performance on offline datasets.
method Random Ensemble Mixture (REM) algorithm for Q-learning, trained on DQN replay dataset.
result Offline REM outperforms strong RL baselines and fully trained DQN agent.
New method tackles safe reinforcement learning from offline data.
problem Learn optimal policies from fixed data while adhering to safety constraints.
method Combines offline RL with online optimization to minimize cumulative cost.
result Proves approximate optimality of the approach under certain conditions.
Fine-tuning RL with offline data reduces online interactions.
problem Optimizing RL with limited online interactions and offline data.
method Developed algorithm extsc{FTPedel} for MDPs with linear structure.
result Optimally reduces the number of online interactions needed.
MOReL learns offline RL policies using pessimistic MDPs.
problem Offline RL's data efficiency and velocity.
method Two-step process: learn P-MDP and near-optimal policy in it.
result MOReL is minimax optimal and matches state-of-the-art results.
Paper offers a fast convergence theory for offline decision making.
problem Offline decision making problems, including reinforcement learning and off-policy evaluation.
method Introduces a framework (DMOF) and algorithm (EDD) with a fast convergence guarantee.
result Demonstrates a fast convergence guarantee with a lower bound complement.
Study finds methods to learn multiple solutions from single task in offline RL.
problem Learning multiple solutions from a single task in offline RL.
method Proposed algorithms for offline RL.
result Empirical results show learning of multiple solutions in offline RL.
This work bridges offline RL and DRL to address distributional shift.
problem Distributional shift in offline RL due to difference in state-action visitation distributions.
method Proposes offline RL algorithms using DRL framework, characterizes sample complexity under single policy concentrability.
result Demonstrates superior performance of proposed algorithms through simulations.
FOCUS improves offline RL by incorporating causal structure into world-models.
problem Learning effective policies from historical data without interaction.
method FOCUS proposes a practical algorithm that learns and leverages causal structure in offline RL.
result FOCUS outperforms plain model-based offline RL algorithms and other causal model-based RL algorithms.
BOMS enhances offline MBRL by improving model selection with Bayesian optimization.
problem Inaccurate model selection in offline MBRL due to distribution shift.
method Proposes BOMS, an active model selection framework using Bayesian optimization.
result Improves model selection with only a small amount of online interaction.
Framework reduces contextual bandit learning to offline regression with near-optimal regret.
problem Efficient learning with large action spaces and complex reward functions.
method Offline Estimation to Decisions (OE2D) algorithm that minimizes regret with near-optimal oracle calls.
result Near-optimal regret for contextual bandits with large action spaces and O(log(T)) offline oracle calls. Paper establishes baselines for offline RL from visual observations.
problem Challenges in offline reinforcement learning from visual observations with continuous action spaces.
method Simple baselines and benchmarking tasks for offline RL from visual observations.
result Simple modifications to existing online RL algorithms outperform existing offline RL methods.
Offline RL with pre-trained features amplifies errors even under mild shifts.
problem Sample-efficient offline RL with pre-trained features under mild distribution shift.
method Empirical study of offline RL with pre-trained neural representations.
result Substantial error amplification occurs even with pre-trained features, requiring stronger conditions for successful offline RL.
Study minimax-optimal rates for offline decision-making with function approximation.
problem Statistical complexity of offline decision-making with function approximation.
method Near minimax-optimal rates for stochastic contextual bandits and Markov decision processes, using pseudo-dimension and behavior policy.
result Established performance limits and new characterization of behavior policy.
A new offline RL framework unifies imitation learning and vanilla offline RL.
problem Learning from expert datasets without active data collection.
method A new offline RL framework that interpolates between imitation learning and vanilla offline RL, using a weak concentrability coefficient and a lower confidence bound algorithm.
result LCB algorithm achieves a faster rate of 1/N for nearly-expert datasets, and is adaptively optimal for the entire data composition range. Anchor-TS uses median anchoring to improve online decision-making from offline data with distribution shift.
problem Improving online decision-making from offline data with distribution shift.
method Sample-Mean Anchored Thompson Sampling (Anchor-TS) with median anchoring.
result Anchor-TS safely leverages offline data to accelerate online learning and reduces regret.
Offline RL policies should adapt to unknown aspects of the environment.
problem Uncertainty in offline RL datasets leads to suboptimal policies.
method Adaptive policies that consider all transitions seen so far, solving an implicit POMDP.
result Optimal adaptive policies improve offline RL performance.
New algorithm for offline RL with linear approx in MDPs and MGs, nearly optimal.
problem Offline RL with linear function approximation in MDPs and MGs.
method Pessimism-based algorithm with uncertainty decomposition via reference function.
result Nearly minimax optimal performance in offline RL for MDPs and MGs.
Safe offline RL for chemical reactors using input convex neural networks.
problem Safe control of exothermic polymerization reactors using historical data.
method Gymnasium-compatible simulation, behaviour cloning, implicit Q-learning, input convex neural networks (PICNNs).
result Offline RL with convex action correction outperforms traditional control approaches.
This paper bridges offline and online RL by studying policy finetuning with a reference policy.
problem Sample-efficient reinforcement learning in online and offline settings.
method Design of policy finetuning algorithms and analysis of sample complexity.
result Theoretical analysis shows that the optimal policy finetuning algorithm is either offline reduction or purely online RL.
Survey of offline RL theory and practical algorithm design challenges.
problem Optimizing return from fixed agent trajectories without additional interactions.
method Theoretical insights and practical algorithm design.
result Conditions for practical offline RL algorithms and their limitations.
A3RL combines online and offline RL with active sampling to improve policy learning.
problem Combining online and offline RL for sample efficiency and robustness.
method A3RL uses a confidence-aware Active Advantage Aligned (A3) sampling strategy to prioritize data from both online and offline sources.
result A3RL outperforms competing online RL techniques that use offline data.
A simple approach to offline RL without additional complexity.
problem Learning from a fixed dataset of actions with value estimation errors.
method Adding a behavior cloning term to the policy update of an online RL algorithm and normalizing the data.
result Matches the performance of state-of-the-art offline RL algorithms with minimal changes.
New assumptions and algorithm solve offline two-player zero-sum Markov games.
problem Solving offline two-player zero-sum Markov games under insufficient assumptions.
method Proposed unilateral concentration assumption and pessimism-type algorithm.
result Algorithm efficiently learns Nash equilibrium under unilateral concentration.
New framework converts offline to online estimation using black-box offline estimators.
problem Convert offline estimation algorithms to online estimation algorithms.
method Oracle-Efficient Online Estimation (OEOE) framework.
result Achieves near-optimal online estimation error via black-box offline estimators.
DeepCO uses deep learning for offline combinatorial optimization in warehouse operations.
problem Optimizing warehouse operation sequences in offline settings.
method DeepCO framework utilizing distribution regularized optimization for TSP.
result DeepCO reduces route length by 5.7% on average for TSP problems.
PASTA optimizes assortment selection using pessimism principle.
problem Optimizing assortment selection with limited data coverage.
method Pessimistic Assortment Optimization (PASTA) based on the principle of pessimism.
result PASTA correctly identifies optimal assortment with minimal data coverage.
Offline RL struggles with sample efficiency due to fundamental barriers.
problem Sample efficiency in offline RL with value function approximation.
method Analyzes the necessity of distributional and representational assumptions.
result Even with concentrability and realizability, sample complexity is polynomial in state space size.
New offline RL study shows exponential sample requirement for accurate policy evaluation.
problem Understanding statistical limits of offline RL with linear function approximation.
method Analyzes necessary representational and distributional conditions for sample-efficient offline reinforcement learning.
result Even with realizability and good feature coverage, offline RL requires exponential samples for accurate policy evaluation.
Hybrid RL algorithm combines offline and online data for robust and efficient policy learning.
problem Combining robust on-policy methods with efficient offline data for hybrid RL.
method Integrates off-policy training on offline data into on-policy NPG framework.
result Achieves state-of-the-art theoretical guarantees and maintains on-policy NPG guarantees.
Model-Based Offline Planning (MBOP) learns models from offline data to control systems directly.
problem Training RL policies from offline data without direct system interaction.
method Generates models from offline data and uses planning to control the system.
result Near-optimal policies found for simulated systems with minimal real-time interaction.