On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,742 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “offline observation”

Challenges and Opportunities in Offline Reinforcement Learning from Visual Observationscs.LG

Paper establishes baselines for offline RL from visual observations.

problem Challenges in offline reinforcement learning from visual observations with continuous action spaces.

method Simple baselines and benchmarking tasks for offline RL from visual observations.

result Simple modifications to existing online RL algorithms outperform existing offline RL methods.

Offline Reinforcement Learning Visual Observations Continuous Control DreamerV2 DrQ-v2

2022-06-09abs ↗pdf ↗

Provably Efficient Third-Person Imitation from Offline Observationcs.LG

Guarantees for third-person imitation learning from offline data.

problem Improving generalizability in imitation learning.

method Problem-dependent statistical learning guarantees for third-person imitation from offline observation.

result Strong performance guarantees for transferred policies in the offline setting.

imitation learning domain adaptation third-person offline observation statistical guarantees

2020-02-27abs ↗pdf ↗

Neural Laplace Control for Continuous-time Delayed Systemscs.LG

Neural Laplace Control tackles offline RL for continuous-time delayed systems with irregular observations.

problem Offline reinforcement learning problems involving continuous-time environments with delays and irregular observations.

method Combines a Neural Laplace dynamics model with a model predictive control (MPC) planner.

result Achieves near expert policy performance on continuous-time delayed environments.

offline reinforcement learning continuous-time systems model-based RL model predictive control neural dynamics

2023-02-24abs ↗pdf ↗

RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcs.LG

RORL improves offline RL robustness with conservative smoothing.

problem Distribution shift and robustness issues in offline RL.

method RORL introduces regularization and conservative smoothing for robustness.

result RORL achieves state-of-the-art performance and robustness to adversarial perturbations.

offline reinforcement learning robustness conservative smoothing reinforcement learning robustness

2022-06-06abs ↗pdf ↗

Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processescs.LG

New algorithm tackles confounding in offline RL for partially observable MDPs.

problem Confounding in offline reinforcement learning for partially observable MDPs.

method P3O algorithm using proximal causal inference and pessimistic confidence regions.

result Achieves

n^{-1/2}

-suboptimality for offline RL in POMDPs with confounded data.

offline reinforcement learning confounding POMDPs partial coverage minimax estimation

2022-05-26abs ↗pdf ↗

Instrumental Variable Value Iteration for Causal Offline Reinforcement Learningstat.ML

A new algorithm uses IVs to learn optimal policies from observational data.

problem Learning optimal policies from unobserved variable confounded data.

method IV-aided Value Iteration (IVVI) algorithm based on conditional moment restrictions.

result First provably efficient algorithm for instrument-aided offline RL.

Instrumental Variables Offline Reinforcement Learning Value Iteration Markov Decision Processes Conditional Moment Restrictions

2021-02-19abs ↗pdf ↗

A Unified Framework of Policy Learning for Contextual Bandit with Confounding Bias and Missing Observationscs.LG

A new algorithm CAP learns optimal policies from observational data with confounding bias and missing observations.

problem Offline contextual bandit with confounding bias and missing observations.

method CAP policy learning, forming reward function as solution of integral equation system, building confidence set, and greedily taking action with pessimism.

result Developed an upper bound to the suboptimality of CAP for the offline contextual bandit problem.

Contextual Bandit Confounding Bias Missing Observations Policy Learning Offline Reinforcement Learning

2023-03-20abs ↗pdf ↗

Identifying Latent Actions and Dynamics from Offline Data via Demonstrator Diversitycs.LG

Identifies latent actions and dynamics from offline data with diverse demonstrators.

problem Recovering latent actions and environment dynamics from action-free trajectories.

method Assumes distinct policies for each demonstrator, identifies latent transitions and policies via matrix factorization.

result Identifies latent transitions and demonstrator policies up to permutation.

Offline Reinforcement Learning Latent Actions Demonstrator Diversity Matrix Factorization Nonnegative Matrix Factorization

2026-03-18abs ↗pdf ↗

DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPscs.LG

DeepAveragers solves offline RL by solving derived MDPs from static data.

problem Offline reinforcement learning with limited data.

method Solves derived non-parametric MDPs (DAC-MDPs) using deep representations and costs for under-represented parts.

result The approach can lower-bound performance and scale to complex offline RL problems.

offline reinforcement learning non-parametric MDPs deep representations costs performance bounds

2020-10-18abs ↗pdf ↗

Offline RL With Resource Constrained Online Deploymentcs.LG

Offline RL tackles resource-constrained online deployment with improved policy transfer.

problem Training policies with limited online features using a rich offline dataset.

method Introduce a policy transfer algorithm that first trains a teacher agent with full offline features and then transfers knowledge to a student agent with limited online features.

result Consistent improvement in performance over baseline methods on resource-constrained datasets.

Offline RL Policy Transfer Resource Constrained D4RL RC-D4RL

2021-10-07abs ↗pdf ↗

Hyperparameter Selection for Offline Reinforcement Learningcs.LG

New method selects best offline RL policies from logged data.

problem Hyperparameter selection challenges offline RL.

method Offline hyperparameter selection for RL algorithms.

result Reliable ranking and selection of policies across hyperparameters.

offline reinforcement learning hyperparameter selection RL algorithms policy ranking log data

2020-07-17abs ↗pdf ↗

Exploiting Generalization in Offline Reinforcement Learning via Unseen State Augmentationscs.LG

New method finds unseen states for RL, improving performance.

problem Offline RL struggles with unseen states and actions.

method Value-informed state perturbations and filtering.

result Improved performance in offline RL tasks.

Offline RL Unseen states Value-informed perturbations Epistemic uncertainty Conservative Q-values

2023-08-07abs ↗pdf ↗

Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Datastat.ML

PESCAL uses mediators to learn from confounded offline data.

problem Learning from confounded observational data in reinforcement learning.

method PESCAL uses mediator variables and the pessimistic principle to address confounding bias and distributional shift.

result It is sufficient to learn a lower bound of the mediator distribution function to mitigate distributional shift.

reinforcement learning observational data confounding bias mediator variable pessimistic principle

2024-03-18abs ↗pdf ↗

Trajectory-Level Data Augmentation for Offline Reinforcement Learningcs.LG

Enhances reinforcement learning from sparse data.

problem Limited data for offline reinforcement learning.

method Trajectory-based data augmentation.

result Improves reinforcement learning performance.

reinforcement learning data augmentation trajectory offline learning suboptimal trajectories

2026-05-13abs ↗pdf ↗

Contextual Inverse Optimization: Offline and Online Learningcs.LG

The paper addresses contextual optimization problems with feedback, aiming to minimize regret.

problem Contextual optimization with feedback information.

method Characterizing the optimal minimax policy in offline setting and leveraging geometric characterization in online setting to optimize cumulative regret.

result Developed an algorithm yielding logarithmic regret bound in the online setting.

contextual optimization feedback regret minimization offline learning online learning

2021-06-26abs ↗pdf ↗

Regret minimization in Linear Bandits with offline data via extended D-optimal explorationcs.LG

Algorithm reduces online regret by leveraging offline data in linear bandits.

problem Online regret minimization in linear bandits with offline data.

method OOPE algorithm using extended D-optimal design.

result Substantial reduction in online regret compared to prior work.

regret minimization linear bandits offline data Frank-Wolfe approximation D-optimal design

2025-08-11abs ↗pdf ↗

Learning an Optimal Assortment Policy under Observational Datastat.ML

Study optimal product assortment using historical data, proving item coverage suffices.

problem Offline assortment optimization under MNL model with limited historical data.

method Pessimistic Rank-Breaking (PRB) algorithm combining rank-breaking and pessimistic estimation.

result Optimal item coverage is both sufficient and necessary for efficient offline learning.

assortment optimization MNL model historical data rank-breaking pessimistic estimation

2025-02-10abs ↗pdf ↗

Offline Learning from Demonstrations and Unlabeled Experiencecs.LG

ORIL learns a reward function from unlabeled data to improve robot learning.

problem Leveraging unlabeled data for robot learning.

method ORIL learns a reward function from demonstrator and unlabeled trajectories, annotates data, and trains an agent via offline reinforcement learning.

result ORIL consistently outperforms BC agents on various robotic tasks.

robot learning offline reinforcement learning unlabeled data behavior cloning reward function

2020-11-27abs ↗pdf ↗

What are the Statistical Limits of Offline RL with Linear Function Approximation?cs.LG

New offline RL study shows exponential sample requirement for accurate policy evaluation.

problem Understanding statistical limits of offline RL with linear function approximation.

method Analyzes necessary representational and distributional conditions for sample-efficient offline reinforcement learning.

result Even with realizability and good feature coverage, offline RL requires exponential samples for accurate policy evaluation.

Offline Reinforcement Learning Statistical Limits Linear Function Approximation Exponential Sample Requirement Policy Evaluation

2020-10-22abs ↗pdf ↗

Offline Data Enhanced On-Policy Policy Gradient with Provable Guaranteescs.LG

Hybrid RL algorithm combines offline and online data for robust and efficient policy learning.

problem Combining robust on-policy methods with efficient offline data for hybrid RL.

method Integrates off-policy training on offline data into on-policy NPG framework.

result Achieves state-of-the-art theoretical guarantees and maintains on-policy NPG guarantees.

Hybrid RL Policy Gradient Offline Data On-Policy Theoretical Guarantees

2023-11-14abs ↗pdf ↗

Bandits with Partially Observable Confounded Datacs.LG

Study linear contextual bandits with confounded offline data, improving regret bounds.

problem Linear contextual bandits with confounded offline data.

method Construct a linear bandit algorithm that utilizes projected information.

result Proved regret bounds that improve current bounds by a factor related to visible dimensionality.

bandits confounded data contextual regret bounds linear

2020-06-11abs ↗pdf ↗

On the Exponential Convergence for Offline RLHF with Pairwise Comparisonscs.LG

RL-LOW algorithm achieves exponential simple regret in offline RLHF with pairwise comparisons.

problem Offline reinforcement learning from human feedback with pairwise comparisons.

method Proposes RL-LOW algorithm to minimize simple regret with exponential convergence.

result Achieves an exponential form of simple regret of $\exp ( - Ω(n/H) )$.

offline reinforcement learning human feedback pairwise comparisons simple regret exponential convergence

2024-06-18abs ↗pdf ↗

Robustly Improving Bandit Algorithms with Confounded and Selection Biased Offline Data: A Causal Approachcs.LG

The paper tackles bandit problems with biased offline data by using causal methods.

problem Improving bandit algorithms with biased offline data that includes confounding and selection biases.

method Formalizes the problem from a causal perspective, categorizes biases, and derives robust bounds for each arm.

result Causal bounds can guide the bandit agent to learn a nearly-optimal decision policy and consistently reduce asymptotic regret.

bandit algorithms biased offline data confounding bias selection bias causal approach

2023-12-20abs ↗pdf ↗

A Policy Gradient Method for Confounded POMDPsstat.ML

Paper proposes a policy gradient method for confounded POMDPs.

problem Estimating policy gradients for confounded POMDPs with continuous state and observation spaces.

method Developed a novel identification result to estimate policy gradients using offline data, solved conditional moment restrictions, and applied min-max learning with function approximation.

result Showed global convergence of the proposed algorithm in finding the optimal policy.

policy gradient confounded POMDPs offline learning min-max learning conditional moment restrictions

2023-05-26abs ↗pdf ↗

MOPO: Model-based Offline Policy Optimizationcs.LG

MOPO optimizes offline RL by penalizing dynamics uncertainty.

problem Learning policies from offline data with distributional shift.

method Modify model-based RL to avoid distributional shift.

result MOPO outperforms model-free and standard model-based RL.

reinforcement learning offline RL distributional shift model-based policy optimization

2020-05-27abs ↗pdf ↗

Provably Efficient Causal Reinforcement Learning with Confounded Observational Datacs.LG

DOVI improves reinforcement learning with offline data, reducing trial-and-error in critical scenarios.

problem Lack of sample efficiency in deep reinforcement learning for critical applications.

method Proposes DOVI algorithm to incorporate confounded observational data provably efficiently.

result DOVI reduces regret by a multiplicative factor compared to pure online setting, especially when data are informative.

reinforcement learning observational data confounding bias sample efficiency causal inference

2020-06-22abs ↗pdf ↗

Provable Efficient Online Matrix Completion via Non-convex Stochastic Gradient Descentcs.LG

Matrix completion, where we wish to recover a low rank matrix by observing a few entries from it, is a widely studied problem in both theory and practice with wide applications. Most of the provable algorithms so far on this problem have been restricted to the offline setting where they provide an estimate of the unkno…

2016-05-26abs ↗pdf ↗

Domain Generalization for Robust Model-Based Offline Reinforcement Learningcs.LG

New RL method handles diverse human operator data.

problem Handling data from multiple, unknown policies.

method Domain-Invariant Model-based Offline RL (DIMORL) with Risk Extrapolation (REx).

result Models trained with REx generalize better across different demonstrators.

offline reinforcement learning domain generalization risk extrapolation multi-demonstrator model-based RL

2022-11-27abs ↗pdf ↗

Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confoundersstat.ML

New method for robust policy evaluation in offline reinforcement learning with sequentially exogenous unobserved confounders.

problem Offline reinforcement learning in domains with unobserved confounders.

method Orthogonalized robust fitted-Q-iteration with closed-form solutions and bias-correction.

result Effective in simulations and real-world data, improving robustness and computational ease.

reinforcement learning offline policy evaluation unobserved confounders sensitivity analysis orthogonalization

2023-02-01abs ↗pdf ↗

A Complete Characterization of Linear Estimators for Offline Policy Evaluationcs.LG

This work characterizes conditions for offline policy evaluation in reinforcement learning.

problem Understanding when classical methods succeed in offline policy evaluation for linear function approximation.

method Control-theoretic and linear-algebraic conditions for classical methods (FQI and LSTD).

result A precise hierarchy of regimes under which these estimators succeed, and a complete picture of their behavior.

Offline Policy Evaluation Reinforcement Learning Linear Estimators Fitted Q-Iteration Least Squares Temporal Difference

2022-03-08abs ↗pdf ↗

Federated Offline Policy Learningcs.LG

The paper tackles personalized policy learning from diverse data sources in a federated setting.

problem Learning personalized decision policies from observational bandit feedback across multiple heterogeneous data sources.

method Introduces a novel regret analysis for distinguishing global and local regret, and presents a federated policy learning algorithm using local policies trained with doubly robust offline policy evaluation strategies.

result Establishes finite-sample upper bounds on global and local regret, characterizing them by source heterogeneity and distribution shift.

policy learning federated learning observational bandit regret analysis heterogeneous data

2023-05-21abs ↗pdf ↗

Overcoming Model Bias for Robust Offline Deep Reinforcement Learningcs.LG

MOOSE improves offline RL robustness by using dynamics models.

problem Low robustness of model-free offline RL algorithms in industrial settings.

method MOOSE uses dynamics models to assess policy performance, keeping policies within data support.

result MOOSE outperforms state-of-the-art model-free offline RL algorithms in robust performance.

Offline RL Model-free Robustness Dynamics models Policy performance

2020-08-12abs ↗pdf ↗

Provable Offline Preference-Based Reinforcement Learningcs.LG

Paper tackles offline preference-based RL with human feedback.

problem Offline Preference-based Reinforcement Learning with preference feedback.

method Two-step approach: MLE for reward estimation and distributionally robust planning.

result First guarantee for learning any target policy with polynomial samples.

Offline RL Preference-based Distributionally robust Maximum Likelihood Estimation Concentrability coefficient

2023-05-24abs ↗pdf ↗

Online Pricing with Offline Data: Phase Transition and Inverse Square Lawcs.LG

This paper investigates the impact of pre-existing offline data on online learning, in the context of dynamic pricing. We study a single-product dynamic pricing problem over a selling horizon of $T$ periods. The demand in each period is determined by the price of the product according to a linear demand model with unkn…

2019-10-19abs ↗pdf ↗

RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learningcs.LG

RL Unplugged benchmarks offline RL methods across diverse domains.

problem Evaluate offline reinforcement learning methods without online data collection.

method Proposes a benchmark suite with diverse datasets and detailed evaluation protocols.

result Demonstrates the effectiveness of offline RL methods across various domains.

reinforcement learning offline methods benchmark RL Unplugged supervised learning

2020-06-24abs ↗pdf ↗

HOFLON: Hybrid Offline Learning and Online Optimization for Process Start-Up and Grade-Transition Controlcs.LG

HOFLON automates process start-ups and grade-changes using offline RL and online optimization.

problem Manual operation of start-ups and grade-changes by experts is declining, leaving plant owners without the necessary tacit know-how.

method HOFLON combines offline RL to learn a latent manifold and long-horizon Q-critic, and online optimization to maximize Q-critic while penalizing deviations and excessive variable changes.

result HOFLON outperforms standard offline RL in industrial case studies, delivering better cumulative rewards than historical data.

process control start-up grade-change reinforcement learning offline learning

2025-10-04abs ↗pdf ↗

Robust Offline Policy Learning with Observational Data from Multiple Sourcescs.LG

The paper tackles robust policy learning from multiple data sources.

problem Learning a policy that generalizes across diverse settings from multiple heterogeneous data sources.

method Proposes a minimax regret optimization objective and a policy learning algorithm combining doubly robust offline policy evaluation and no-regret learning.

result Achieves minimal worst-case mixture regret up to a moderated vanishing rate of the total data across all sources.

policy learning observational data minimax regret heterogeneous sources doubly robust

2024-10-11abs ↗pdf ↗

Online Learning via Offline Greedy Algorithms: Applications in Market Design and Optimizationcs.LG

Transforms offline greedy algorithms to online algorithms for combinatorial problems.

problem Online decision-making in time-varying combinatorial environments.

method General framework using Blackwell approachability and Bandit Blackwell approachability.

result Achieves

O(\sqrt{T})

regret in full information setting and

O(T^{2/3})

regret in bandit setting.

Online Learning Greedy Algorithms Blackwell Approachability Bandit Blackwell Approachability Regret Analysis

2021-02-18abs ↗pdf ↗

LazyDINO: Fast, scalable, and efficiently amortized Bayesian inversion via structure-exploiting and surrogate-driven measure transportmath.NA

LazyDINO efficiently solves high-dimensional Bayesian inverse problems with fast and scalable solutions.

problem High-dimensional nonlinear Bayesian inverse problems with expensive parameter-to-observable maps.

method LazyDINO combines derivative-informed neural surrogates and lazy map variational inference for efficient posterior approximation.

result Significant cost reduction in amortized Bayesian inversion, achieving one to two orders of magnitude improvement.

Bayesian Inverse Problems Transport Maps Surrogate Models Lazy Maps Amortized Inference

2024-11-19abs ↗pdf ↗

Diffusion Self-Weighted Guidance for Offline Reinforcement Learningcs.LG

A new method for offline RL using diffusion models and self-weighted guidance.

problem Challenges in computing scores for offline RL using weight functions.

method Constructing a diffusion over actions and weights, using the diffusion model for guidance.

result Performs on par with state-of-the-art methods on challenging environments.

Offline Reinforcement Learning Diffusion Models Self-Weighted Guidance Reinforcement Learning Weight Functions

2025-05-23abs ↗pdf ↗

A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscs.LG

Survey on offline RL methods for real-world applications.

problem Inaccessible domains due to high cost and danger of interacting with the environment.

method Unifying taxonomy and review of offline RL algorithms.

result Wide range of applications for offline RL, including education, healthcare, and robotics.

reinforcement learning offline learning real-world applications taxonomy benchmarking

2022-03-02abs ↗pdf ↗

Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcs.LG

The paper proposes autoregressive models for better offline RL.

problem Offline RL policy evaluation and optimization challenges.

method Autoregressive dynamics models for sequential state and reward prediction.

result Autoregressive models outperform standard methods in log-likelihood and RL tasks.

offline RL autoregressive models policy evaluation policy optimization

2021-04-28abs ↗pdf ↗

Offline Contextual Bandits with Overparameterized Modelscs.LG

Overparameterized models generalize well in offline contextual bandits, but policy-based algorithms struggle.

problem The performance gap between value-based and policy-based algorithms in offline contextual bandits with overparameterized models.

method Analysis of action-stability in objectives and formal proofs of regret bounds.

result The performance gap is due to action-stability of objectives, with value-based objectives being stable and policy-based objectives unstable.

offline contextual bandits overparameterized models action-stability regret bounds value-based algorithms

2020-06-27abs ↗pdf ↗

Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcs.LG

Transformers learn to make decisions in new contexts from offline data.

problem Understanding when and how transformers can perform in-context reinforcement learning.

method Theoretical framework analyzing supervised pretraining for ICRL, including algorithm distillation and decision-pretrained transformers.

result Transformers can efficiently approximate optimal reinforcement learning algorithms for various environments.

transformers in-context reinforcement learning supervised pretraining algorithm distillation decision-pretrained transformers

2023-10-12abs ↗pdf ↗

Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Rewardcs.LG

A new algorithm for offline RL with trajectory-wise reward reduces bias and variance errors.

problem Offline RL with trajectory-wise reward incurs large bias and variance errors.

method PARTED algorithm that decomposes trajectory return into proxy rewards and performs pessimistic value iteration.

result PARTED achieves provably efficient suboptimality bounds in general MDPs with trajectory-wise reward.

offline reinforcement learning trajectory-wise reward value iteration suboptimality bounds reinforcement learning algorithms

2022-06-13abs ↗pdf ↗

The Value of Reward Lookahead in Reinforcement Learningcs.LG

This work analyzes the value of future reward information in RL.

problem Analyzing the impact of knowing future rewards in reinforcement learning.

method Competitive analysis and worst-case reward distribution.

result Exact ratios between standard RL agents and those with future-reward lookahead.

reinforcement learning competitive analysis future rewards reward lookahead offline RL

2024-03-18abs ↗pdf ↗

The Power of Batching in Multiple Hypothesis Testingstat.ME

One important partition of algorithms for controlling the false discovery rate (FDR) in multiple testing is into offline and online algorithms. The first generally achieve significantly higher power of discovery, while the latter allow making decisions sequentially as well as adaptively formulating hypotheses based on …

2019-10-11abs ↗pdf ↗

Leveraging Optimal Transport for Enhanced Offline Reinforcement Learning in Surgical Robotic Environmentscs.AI

Paper introduces OTR for efficient offline RL in surgical robotics.

problem Lack of annotated datasets for offline RL in surgical robotics.

method OTR algorithm using Optimal Transport to assign rewards to unlabeled trajectories.

result OTR enables efficient policy learning from large datasets without handcrafted rewards.

Optimal Transport Offline Reinforcement Learning Surgical Robotics Reward Assignment Reinforcement Learning

2023-10-13abs ↗pdf ↗