Pessimistic estimator improves multi-objective policy optimization.
problem Optimizing multi-objective policies from existing data.
method Pessimistic estimator based on inverse propensity scores (IPS).
result Pessimistic estimator outperforms naive IPS estimator in theory and experiments.
This paper develops a model of reference-dependent assessment of subjective beliefs in which loss-averse people optimally choose the expectation as the reference point to balance the current felicity from the optimistic anticipation and the future disappointment from the realisation. The choice of over-optimism or over…
A new approach estimates propagators for trading risky assets.
problem Estimating price impact kernel from static data for optimal trading.
method Nonparametric estimation of propagator using offline reinforcement learning.
result Pessimistic trading strategy optimises execution costs under uncertainty.
Pessimistic Q-learning improves sample efficiency in offline reinforcement learning.
problem Insufficient coverage and sample scarcity in offline reinforcement learning datasets.
method Pessimistic Q-learning algorithm for offline reinforcement learning, focusing on variance reduction.
result Near-optimal sample complexity achieved with the proposed algorithm.
Study proposes a new risk measure for optimal portfolio allocation.
problem Challenges in estimating optimal portfolios based on pessimistic risk.
method Introduces uniform pessimistic risk and computational algorithm.
result Demonstrates the usefulness of the proposed risk and portfolio model with real data analysis.
Consider a classification problem where we have both labeled and unlabeled data available. We show that for linear classifiers defined by convex margin-based surrogate losses that are decreasing, it is impossible to construct any semi-supervised approach that is able to guarantee an improvement over the supervised clas…
New algorithm reduces high-probability regret for time-varying feedback graphs.
problem High-probability regret bounds for adversarial bandits with time-varying feedback graphs.
method Online mirror descent framework with innovative techniques for pessimistic loss estimators.
result Achieves optimal high-probability regret bound for general and weakly observable graphs.
Proposes Optimistic Pessimistically Initialised Q-Learning (OPIQ) for better exploration in RL.
problem Pessimistic initialisation of Q-values in deep RL leads to poor exploration performance.
method Augments pessimistically initialised Q-values with count-based bonuses to ensure optimism.
result OPIQ outperforms non-optimistic DQN variants in hard exploration tasks.
This paper explains why distributional reinforcement learning is better than vanilla RL using small-loss bounds.
problem Understanding when and why distributional reinforcement learning (DistRL) is superior to vanilla reinforcement learning (RL).
method The paper uses small-loss bounds to explain the benefits of DistRL, proposing algorithms and proving bounds for different RL settings.
result Distributional reinforcement learning (DistRL) outperforms vanilla RL when optimal costs are small, as shown by small-loss bounds.
Paper develops neural network approximation for pessimistic offline RL with theoretical guarantees.
problem Challenges in offline reinforcement learning with deep neural networks and data dependence.
method Establishes estimation error for pessimistic offline RL using neural network approximation with C \mathcal{C} C -mixing data. result Explicit efficiency of deep adversarial offline RL frameworks demonstrated with two converging error components.
Study of insurance market equilibria with risk-averse policyholders.
problem Analyzing optimal insurance contracts in a monopoly market with risk-averse policyholders.
method Modeling Stackelberg equilibria with a profit-maximizing insurer and a risk-averse policyholder.
result Equilibrium contracts exhibit a layer-type structure, providing full insurance over pessimistic loss layers and no coverage over optimistic ones.
Improved DPO framework penalizes preference uncertainty to avoid overoptimization.
problem Aligning LLMs to human preferences is challenging due to varied, context-dependent, and ambiguous preferences.
method Developed a pessimistic framework for DPO by introducing preference uncertainty penalization schemes.
result Improved overall performance and better completions on high-uncertainty responses compared to vanilla DPO.
Study optimal product assortment using historical data, proving item coverage suffices.
problem Offline assortment optimization under MNL model with limited historical data.
method Pessimistic Rank-Breaking (PRB) algorithm combining rank-breaking and pessimistic estimation.
result Optimal item coverage is both sufficient and necessary for efficient offline learning.
PESCAL uses mediators to learn from confounded offline data.
problem Learning from confounded observational data in reinforcement learning.
method PESCAL uses mediator variables and the pessimistic principle to address confounding bias and distributional shift.
result It is sufficient to learn a lower bound of the mediator distribution function to mitigate distributional shift.
Pessimistic Minimax Value Iteration finds efficient NE policies from offline data.
problem Finding an approximate Nash equilibrium in offline Markov games with non-uniform coverage.
method Pessimistic Minimax Value Iteration (PMVI) constructs pessimistic value function estimates and solves NEs.
result Established a nearly minimax optimal result for offline Markov games with function approximation.
POLAR optimizes treatment strategies in dynamic settings with statistical guarantees.
problem Optimizing sequential decisions in dynamic treatment regimes with robustness and statistical guarantees.
method Pessimistic model-based approach estimating transition dynamics and incorporating uncertainty penalties.
result Offers statistical and computational guarantees, including finite-sample bounds on policy suboptimality.
Proposes DRRO to mitigate over-optimization in RLHF from human feedback.
problem Over-optimization due to reward misspecification in RLHF.
method Wasserstein distributionally robust regret optimization (DRRO).
result DRRO mitigates over-optimization more effectively than existing baselines.
This work improves policy evaluation and selection using logarithmic smoothing for pessimistic off-policy estimation.
problem Offline evaluation and selection of policies from past data.
method Develops novel concentration bounds and a logarithmically smoothed estimator (LS) for improved policy selection and learning.
result The logarithmically smoothed estimator (LS) provides tighter bounds and better policy selection and learning.
New method estimates policy performance under unobserved confounding.
problem Estimating policy performance when decisions depend on unobserved variables.
method Developed worst-case bounds for robust OPE under unobserved confounding.
result Efficient procedure for computing worst-case bounds, proving statistical consistency.
Papers learn from data to make decisions without interacting, improving on previous methods.
problem Achieving optimal decision-making from offline data with non-linear function approximation.
method Pessimistic Nonlinear Least-Square Value Iteration (PNLSVI) with three innovative components.
result Achieves minimax optimal instance-dependent regret for non-linear function approximation.
Study online learning with off-policy feedback in adversarial bandit problems.
problem Learning with limited direct feedback in sequential decision making.
method Proposed algorithms that adapt pessimistic reward estimators to handle unknown behavior policy.
result Guaranteed regret bounds scaling with policy mismatch, improving performance against well-covered comparators.
Approximate dynamic programming is a popular method for solving large Markov decision processes. This paper describes a new class of approximate dynamic programming (ADP) methods- distributionally robust ADP-that address the curse of dimensionality by minimizing a pessimistic bound on the policy loss. This approach tur…
New algorithm for reinforcement learning in uncertain environments with unknown thresholds.
problem Safety in reinforcement learning in unknown and uncertain environments.
method Growing-Window estimator sampling and Stochastic Pessimistic-Optimistic Thresholding (SPOT) algorithm.
result Achieves sublinear regret and constraint violation of i l d e O ( T ) ilde{\mathcal{O}}(\sqrt{T}) i l d e O ( T ) . Paper tackles constrained bandit problems with a new learning framework.
problem Optimizing a black-box reward function subject to a black-box constraint function over a continuous space.
method Rectified Pessimistic-Optimistic Learning (RPOL) framework, incorporating optimistic and pessimistic GP bandit learning.
result RPOL achieves sublinear regret and minimal cumulative constraint violation.
The paper analyzes RLHF with human feedback and provides convergence results for MLE and pessimistic MLE.
problem Improving RLHF with human feedback from pairwise or K K K -wise comparisons. method Theoretical framework for RLHF with convergence analysis of MLE and pessimistic MLE.
result MLE fails but pessimistic MLE provides improved policies under certain coverage assumptions.
We introduce novel variants of momentum by incorporating the variance of the stochastic loss function. The variance characterizes the confidence or uncertainty of the local features of the averaged loss surface across the i.i.d. subsets of the training data defined by the mini-batches. We show two applications of the g…
MOReL learns offline RL policies using pessimistic MDPs.
problem Offline RL's data efficiency and velocity.
method Two-step process: learn P-MDP and near-optimal policy in it.
result MOReL is minimax optimal and matches state-of-the-art results.
Study tight offline learning bounds for linear MDPs using variance information.
problem Understanding statistical limits with linear function representations in offline reinforcement learning.
method Variance-aware pessimistic value iteration (VAPVI) that reweights Bellman residuals based on estimated variances.
result Improved offline learning bounds expressed in terms of system quantities.
New algorithm reduces best-in-class regret in contextual bandits.
problem Compete with the best policy in a class without model restrictions.
method Proposes an algorithm that updates policies by minimizing a pessimistic objective, including a clipped inverse-propensity estimate and variance penalty.
result Achieves fast best-in-class regret rates, including polylogarithmic rates in the parametric case.
Improves policy optimization with polylog(T) regret bounds for stochastic losses.
problem Improves theoretical guarantees for policy optimization in stochastic settings.
method Leverages Tsallis and Shannon entropy regularizers for polylog(T) regret, and log-barrier regularizer for adversarial settings.
result Achieves a first-order polylog(T) regret bound for policy optimization in stochastic settings.
Semi-pessimistic RL tackles distributional shift and data scarcity in offline RL.
problem Distributional shift and scarcity of labeled data in offline RL.
method Proposes a semi-pessimistic RL method that simplifies learning by seeking a lower bound of the reward function.
result Demonstrates clear competitiveness and improved policy learning with vast unlabeled data.
Q-Distribution Guided Q-Learning corrects overestimation of uncertain OOD actions in offline RL.
problem Overestimation of Q-values for out-of-distribution actions in offline reinforcement learning.
method QDQ applies a pessimistic adjustment to Q-values in uncertain OOD regions based on a consistency model.
result QDQ improves performance on the D4RL benchmark and achieves significant improvements across many tasks.
This study tackles adversarial corruption in model-based reinforcement learning.
problem Adversarial corruption in model-based reinforcement learning.
method Maximum likelihood estimation (MLE) approach for learning transition model in both online and offline settings.
result Proves a regret of i l d e O ( T + C ) ilde{\mathcal{O}}(\sqrt{T} + C) i l d e O ( T + C ) for CR-OMLE and a suboptimality of O ( C / n ) \mathcal{O}(C/n) O ( C / n ) for CR-PMLE. Distributionally Robust Supervised Learning (DRSL) is necessary for building reliable machine learning systems. When machine learning is deployed in the real world, its performance can be significantly degraded because test data may follow a different distribution from training data. DRSL with f-divergences explicitly …
Dual active learning improves RLHF by selecting optimal conversations and teachers.
problem Efficiently aligning LLMs with human preferences using RLHF from feedback.
method Offline RL for conversation and teacher selection, dual active reward learning, pessimistic RL.
result The proposed algorithm achieves minimal generalized variance and outperforms state-of-the-arts.
New methods tackle robust reinforcement learning in sparse, corrupted data.
problem Tackles robust reinforcement learning in sparse, corrupted data.
method Proposes actor-critic methods with sparse robust estimator oracles.
result First non-vacuous guarantees in high-dimensional sparse MDPs with single-policy concentrability coverage.
PASTA optimizes assortment selection using pessimism principle.
problem Optimizing assortment selection with limited data coverage.
method Pessimistic Assortment Optimization (PASTA) based on the principle of pessimism.
result PASTA correctly identifies optimal assortment with minimal data coverage.
New algorithm achieves small-loss bounds in online learning with improved rates.
problem Achieving strong stability in online learning algorithms.
method Introduces ρ ρ ρ -separation to enforce strong stability, unifying previous approaches. result Oracle-efficient algorithm achieves small-loss bounds with improved rates.
Robust OPE framework uses human inputs to improve policy evaluation in changing environments.
problem Inaccurate policy evaluations due to shifts in environment properties.
method Adapts OPE methods to shifts on user-inputted covariates, providing more realistic utility estimates.
result Robust OPE framework yields less pessimistic policy evaluations and captures realistic dataset shifts.
New algorithm tackles multi-agent reinforcement learning with optimal convergence rate.
problem Multi-agent reinforcement learning with large state spaces and linear function approximations.
method Refined AVLPR framework with data-dependent pessimistic estimation and action-dependent bonuses.
result First algorithm with optimal O ( T − 1 / 2 ) O(T^{-1/2}) O ( T − 1/2 ) convergence rate and no poly( A max A_{\max} A m a x ) dependency. KG-WDRO optimizes transfer learning with external knowledge.
problem Over-pessimism in WDRO for small target samples.
method KG-WDRO incorporates multiple sources of external knowledge to construct smaller Wasserstein ambiguity sets.
result KG-WDRO improves transfer learning performance and adaptivity.
Actor-critic methods, a type of model-free Reinforcement Learning, have been successfully applied to challenging tasks in continuous control, often achieving state-of-the art performance. However, wide-scale adoption of these methods in real-world domains is made difficult by their poor sample efficiency. We address th…
Random Fourier features is a widely used, simple, and effective technique for scaling up kernel methods. The existing theoretical analysis of the approach, however, remains focused on specific learning tasks and typically gives pessimistic bounds which are at odds with the empirical results. We tackle these problems an…
New approach improves adversarial robustness without sacrificing natural generalization.
problem Balancing adversarial robustness and natural generalization in machine learning.
method Friendly adversarial training (FAT) using early-stopped PGD to find least adversarial data.
result Early-stopped PGD achieves adversarial robustness without compromising natural generalization.
It is well known that quantile regression model minimizes the portfolio extreme risk, whenever the attention is placed on the estimation of the response variable left quantiles. We show that, by considering the entire conditional distribution of the dependent variable, it is possible to optimize different risk and perf…
Combines experimental and historical data for robust policy evaluation.
problem Policy evaluation with mixed data sources, especially experimental vs historical.
method Linear integration of estimators from experimental and historical data, optimized for MSE minimization.
result Proposed estimators outperform traditional methods in ridesharing company data.
Portfolio allocation is crucial for investment companies. However, getting the best strategy in a complex and dynamic stock market is challenging. In this paper, we propose a novel Adaptive Deep Deterministic Reinforcement Learning scheme (Adaptive DDPG) for the portfolio allocation task, which incorporates optimistic …
Paper tackles RLHF with DCPPO method, proving near-optimal suboptimality.
problem Challenges in offline RLHF with limited human feedback and bounded rationality.
method DCPPO method involving three stages: MLE, reward function recovery, and pessimistic value iteration.
result DCPPO's suboptimality almost matches classical pessimistic offline RL in terms of distribution shift and dimension.