Pessimistic estimator improves multi-objective policy optimization.
problem Optimizing multi-objective policies from existing data.
method Pessimistic estimator based on inverse propensity scores (IPS).
result Pessimistic estimator outperforms naive IPS estimator in theory and experiments.
Pessimistic Q-learning improves sample efficiency in offline reinforcement learning.
problem Insufficient coverage and sample scarcity in offline reinforcement learning datasets.
method Pessimistic Q-learning algorithm for offline reinforcement learning, focusing on variance reduction.
result Near-optimal sample complexity achieved with the proposed algorithm.
Study proposes a new risk measure for optimal portfolio allocation.
problem Challenges in estimating optimal portfolios based on pessimistic risk.
method Introduces uniform pessimistic risk and computational algorithm.
result Demonstrates the usefulness of the proposed risk and portfolio model with real data analysis.
Proposes Optimistic Pessimistically Initialised Q-Learning (OPIQ) for better exploration in RL.
problem Pessimistic initialisation of Q-values in deep RL leads to poor exploration performance.
method Augments pessimistically initialised Q-values with count-based bonuses to ensure optimism.
result OPIQ outperforms non-optimistic DQN variants in hard exploration tasks.
Paper develops neural network approximation for pessimistic offline RL with theoretical guarantees.
problem Challenges in offline reinforcement learning with deep neural networks and data dependence.
method Establishes estimation error for pessimistic offline RL using neural network approximation with C \mathcal{C} C -mixing data. result Explicit efficiency of deep adversarial offline RL frameworks demonstrated with two converging error components.
Study optimal product assortment using historical data, proving item coverage suffices.
problem Offline assortment optimization under MNL model with limited historical data.
method Pessimistic Rank-Breaking (PRB) algorithm combining rank-breaking and pessimistic estimation.
result Optimal item coverage is both sufficient and necessary for efficient offline learning.
PESCAL uses mediators to learn from confounded offline data.
problem Learning from confounded observational data in reinforcement learning.
method PESCAL uses mediator variables and the pessimistic principle to address confounding bias and distributional shift.
result It is sufficient to learn a lower bound of the mediator distribution function to mitigate distributional shift.
Pessimistic Minimax Value Iteration finds efficient NE policies from offline data.
problem Finding an approximate Nash equilibrium in offline Markov games with non-uniform coverage.
method Pessimistic Minimax Value Iteration (PMVI) constructs pessimistic value function estimates and solves NEs.
result Established a nearly minimax optimal result for offline Markov games with function approximation.
POLAR optimizes treatment strategies in dynamic settings with statistical guarantees.
problem Optimizing sequential decisions in dynamic treatment regimes with robustness and statistical guarantees.
method Pessimistic model-based approach estimating transition dynamics and incorporating uncertainty penalties.
result Offers statistical and computational guarantees, including finite-sample bounds on policy suboptimality.
A new approach estimates propagators for trading risky assets.
problem Estimating price impact kernel from static data for optimal trading.
method Nonparametric estimation of propagator using offline reinforcement learning.
result Pessimistic trading strategy optimises execution costs under uncertainty.
Proposes DRRO to mitigate over-optimization in RLHF from human feedback.
problem Over-optimization due to reward misspecification in RLHF.
method Wasserstein distributionally robust regret optimization (DRRO).
result DRRO mitigates over-optimization more effectively than existing baselines.
This work improves policy evaluation and selection using logarithmic smoothing for pessimistic off-policy estimation.
problem Offline evaluation and selection of policies from past data.
method Develops novel concentration bounds and a logarithmically smoothed estimator (LS) for improved policy selection and learning.
result The logarithmically smoothed estimator (LS) provides tighter bounds and better policy selection and learning.
Papers learn from data to make decisions without interacting, improving on previous methods.
problem Achieving optimal decision-making from offline data with non-linear function approximation.
method Pessimistic Nonlinear Least-Square Value Iteration (PNLSVI) with three innovative components.
result Achieves minimax optimal instance-dependent regret for non-linear function approximation.
Study online learning with off-policy feedback in adversarial bandit problems.
problem Learning with limited direct feedback in sequential decision making.
method Proposed algorithms that adapt pessimistic reward estimators to handle unknown behavior policy.
result Guaranteed regret bounds scaling with policy mismatch, improving performance against well-covered comparators.
New algorithm for reinforcement learning in uncertain environments with unknown thresholds.
problem Safety in reinforcement learning in unknown and uncertain environments.
method Growing-Window estimator sampling and Stochastic Pessimistic-Optimistic Thresholding (SPOT) algorithm.
result Achieves sublinear regret and constraint violation of i l d e O ( T ) ilde{\mathcal{O}}(\sqrt{T}) i l d e O ( T ) . Paper tackles constrained bandit problems with a new learning framework.
problem Optimizing a black-box reward function subject to a black-box constraint function over a continuous space.
method Rectified Pessimistic-Optimistic Learning (RPOL) framework, incorporating optimistic and pessimistic GP bandit learning.
result RPOL achieves sublinear regret and minimal cumulative constraint violation.
The paper analyzes RLHF with human feedback and provides convergence results for MLE and pessimistic MLE.
problem Improving RLHF with human feedback from pairwise or K K K -wise comparisons. method Theoretical framework for RLHF with convergence analysis of MLE and pessimistic MLE.
result MLE fails but pessimistic MLE provides improved policies under certain coverage assumptions.
MOReL learns offline RL policies using pessimistic MDPs.
problem Offline RL's data efficiency and velocity.
method Two-step process: learn P-MDP and near-optimal policy in it.
result MOReL is minimax optimal and matches state-of-the-art results.
Study tight offline learning bounds for linear MDPs using variance information.
problem Understanding statistical limits with linear function representations in offline reinforcement learning.
method Variance-aware pessimistic value iteration (VAPVI) that reweights Bellman residuals based on estimated variances.
result Improved offline learning bounds expressed in terms of system quantities.
Semi-pessimistic RL tackles distributional shift and data scarcity in offline RL.
problem Distributional shift and scarcity of labeled data in offline RL.
method Proposes a semi-pessimistic RL method that simplifies learning by seeking a lower bound of the reward function.
result Demonstrates clear competitiveness and improved policy learning with vast unlabeled data.
Q-Distribution Guided Q-Learning corrects overestimation of uncertain OOD actions in offline RL.
problem Overestimation of Q-values for out-of-distribution actions in offline reinforcement learning.
method QDQ applies a pessimistic adjustment to Q-values in uncertain OOD regions based on a consistency model.
result QDQ improves performance on the D4RL benchmark and achieves significant improvements across many tasks.
This study tackles adversarial corruption in model-based reinforcement learning.
problem Adversarial corruption in model-based reinforcement learning.
method Maximum likelihood estimation (MLE) approach for learning transition model in both online and offline settings.
result Proves a regret of i l d e O ( T + C ) ilde{\mathcal{O}}(\sqrt{T} + C) i l d e O ( T + C ) for CR-OMLE and a suboptimality of O ( C / n ) \mathcal{O}(C/n) O ( C / n ) for CR-PMLE. This paper develops a model of reference-dependent assessment of subjective beliefs in which loss-averse people optimally choose the expectation as the reference point to balance the current felicity from the optimistic anticipation and the future disappointment from the realisation. The choice of over-optimism or over…
Dual active learning improves RLHF by selecting optimal conversations and teachers.
problem Efficiently aligning LLMs with human preferences using RLHF from feedback.
method Offline RL for conversation and teacher selection, dual active reward learning, pessimistic RL.
result The proposed algorithm achieves minimal generalized variance and outperforms state-of-the-arts.
New methods tackle robust reinforcement learning in sparse, corrupted data.
problem Tackles robust reinforcement learning in sparse, corrupted data.
method Proposes actor-critic methods with sparse robust estimator oracles.
result First non-vacuous guarantees in high-dimensional sparse MDPs with single-policy concentrability coverage.
PASTA optimizes assortment selection using pessimism principle.
problem Optimizing assortment selection with limited data coverage.
method Pessimistic Assortment Optimization (PASTA) based on the principle of pessimism.
result PASTA correctly identifies optimal assortment with minimal data coverage.
New algorithm reduces high-probability regret for time-varying feedback graphs.
problem High-probability regret bounds for adversarial bandits with time-varying feedback graphs.
method Online mirror descent framework with innovative techniques for pessimistic loss estimators.
result Achieves optimal high-probability regret bound for general and weakly observable graphs.
Robust OPE framework uses human inputs to improve policy evaluation in changing environments.
problem Inaccurate policy evaluations due to shifts in environment properties.
method Adapts OPE methods to shifts on user-inputted covariates, providing more realistic utility estimates.
result Robust OPE framework yields less pessimistic policy evaluations and captures realistic dataset shifts.
Actor-critic methods, a type of model-free Reinforcement Learning, have been successfully applied to challenging tasks in continuous control, often achieving state-of-the art performance. However, wide-scale adoption of these methods in real-world domains is made difficult by their poor sample efficiency. We address th…
It is well known that quantile regression model minimizes the portfolio extreme risk, whenever the attention is placed on the estimation of the response variable left quantiles. We show that, by considering the entire conditional distribution of the dependent variable, it is possible to optimize different risk and perf…
Combines experimental and historical data for robust policy evaluation.
problem Policy evaluation with mixed data sources, especially experimental vs historical.
method Linear integration of estimators from experimental and historical data, optimized for MSE minimization.
result Proposed estimators outperform traditional methods in ridesharing company data.
Portfolio allocation is crucial for investment companies. However, getting the best strategy in a complex and dynamic stock market is challenging. In this paper, we propose a novel Adaptive Deep Deterministic Reinforcement Learning scheme (Adaptive DDPG) for the portfolio allocation task, which incorporates optimistic …
Paper tackles RLHF with DCPPO method, proving near-optimal suboptimality.
problem Challenges in offline RLHF with limited human feedback and bounded rationality.
method DCPPO method involving three stages: MLE, reward function recovery, and pessimistic value iteration.
result DCPPO's suboptimality almost matches classical pessimistic offline RL in terms of distribution shift and dimension.
Investor expectations shifted pessimistically during the 2020 stock market crash and recovery.
problem Analyzing changes in investor expectations during the 2020 stock market crash and recovery.
method Surveying Vanguard clients at three points: before, during, and after the crash.
result Investor pessimism increased following the crash, with significant disagreement about future outcomes.
Proposes a method to improve pWCET estimation for heavy-tailed distributions.
problem Improving pWCET estimation for heavy-tailed distributions in real-time systems.
method Incorporates saturating functions into Chebyshev's inequality to mitigate the influence of large outliers.
result Achieves safe and tighter bounds for heavy-tailed distributions.
New Bellman error estimator improves offline model selection performance.
problem Selecting the best policy from logged data using mean squared Bellman error.
method Developed a more accurate estimator of MSBE and analyzed conditions for successful OMS.
result New estimator achieves impressive offline model selection performance on diverse tasks.
A new algorithm improves offline reinforcement learning robustness.
problem Finding optimal policies in perturbed environments from offline data.
method Doubly Pessimistic Model-based Policy Optimization (P^2MPO) framework.
result Proves sample efficiency with robust partial coverage data.
Develops methods for dynamic pricing in incomplete data settings.
problem Incomplete historical data makes optimal pricing difficult.
method Nonparametric partial identification framework for offline dynamic pricing.
result Pessimistic and opportunistic policies with regret bounds.
Improvement guarantees for semi-supervised classifiers can currently only be given under restrictive conditions on the data. We propose a general way to perform semi-supervised parameter estimation for likelihood-based classifiers for which, on the full training set, the estimates are never worse than the supervised so…
New algorithm identifies near-optimal policies in adversarial distributed RL settings.
problem Adversarial agents in distributed RL settings that can collude and report arbitrary data.
method Weighted-Clique algorithm for robust mean estimation from batches, combined with novel distributed algorithms.
result Achieves superior robustness guarantees and near-optimal sample complexities in both offline and online settings.
In domain adaptation, classifiers with information from a source domain adapt to generalize to a target domain. However, an adaptive classifier can perform worse than a non-adaptive classifier due to invalid assumptions, increased sensitivity to estimation errors or model misspecification. Our goal is to develop a doma…
LLMs overestimate stock returns and are less accurate at predicting extreme outcomes.
problem Behavioral biases in LLMs' stock return forecasts.
method Comparison of LLM forecasts with crowd-sourced estimates and historical data.
result LLMs overestimate stock returns and are less accurate at predicting extreme outcomes.
A new algorithm for offline RL with trajectory-wise reward reduces bias and variance errors.
problem Offline RL with trajectory-wise reward incurs large bias and variance errors.
method PARTED algorithm that decomposes trajectory return into proxy rewards and performs pessimistic value iteration.
result PARTED achieves provably efficient suboptimality bounds in general MDPs with trajectory-wise reward.
We propose a straightforward extension of our previously proposed log-periodic power law model of the ``anti-bubble'' regime of the USA market since the summer of 2000, in terms of the renormalization group framework to model critical points. Using a previous work by Gluzman and Sornette (2002) on the classification of…
Improved bounds for function approximation in nonlinear sets.
problem Achieving high probability error with limited samples in nonlinear function approximation.
method Restricting model class to a neighbourhood of the best approximation and estimating sample complexity using tangent and normal spaces' complexities and curvature.
result Improved worst-case bounds for sample complexity in more general sets like tensor networks and neural networks.
Efficient offline reinforcement learning with neural networks using differentiable function approximation.
problem Statistical efficiency of offline reinforcement learning with function approximators.
method Pessimistic fitted Q-learning (PFQL) and differentiable function approximation.
result Provably efficient offline reinforcement learning with differentiable function approximation.
The paper introduces Bellman-consistent pessimism to improve offline reinforcement learning without overly pessimistic bias.
problem Offline reinforcement learning's challenge of discovering good policies without exhaustive exploration.
method Introduces Bellman-consistent pessimism for function approximation, improving sample complexity and adaptability.
result Improves sample complexity by O ( d ) \mathcal{O}(d) O ( d ) in the action space finite case, and automatically adapts to bias-variance tradeoff. An algorithm for maximizing rewards under linear cost constraints.
problem Maximizing rewards while adhering to cost constraints in a linear bandit problem.
method Proposes an upper-confidence bound algorithm called optimistic pessimistic linear bandit (OPLB) for constrained contextual linear bandits.
result Proves an O ~ ( d T τ − c 0 ) \widetilde{\mathcal{O}}(\frac{d\sqrt{T}}{τ-c_0}) O ( τ − c 0 d T ) bound on regret for the proposed algorithm.