Paper proposes PRR network for better experience reuse in reinforcement learning.
problem Efficient experience reuse in reinforcement learning across multiple granularities.
method Proposes PRR network trained on multi-level architecture to extract and store experience.
result PRR network leads to better experience reuse and improved performance.
DAGR improves navigation by refining goal representations conditioned on the current state.
problem Goal-conditioned reinforcement learning lacks state awareness, leading to inefficient policy recovery.
method DAGR refines static goal embeddings into state-conditioned ones using gated cross-attention with a state-goal discrepancy map.
result DAGR improves navigation tasks on OGBench, matching or outperforming base methods.
BRPO optimizes batch RL policies to better exploit state-action differences.
problem Batch RL's conservatism limits exploitation of state-action differences.
method Proposes residual policies and derives BRPO to maximize policy performance.
result BRPO achieves state-of-the-art performance in various tasks.
New method for distributional off-policy evaluation using Bellman residual minimization.
problem Learning return distribution from offline data generated by a different policy.
method Energy Bellman Residual Minimizer (EBRM) method.
result Established finite-sample error bound for EBRM estimator.
This paper aims at theoretically and empirically comparing two standard optimization criteria for Reinforcement Learning: i) maximization of the mean value and ii) minimization of the Bellman residual. For that purpose, we place ourselves in the framework of policy search algorithms, that are usually designed to maximi…
New insights into state representations in RL help design better learning rules.
problem Lack of automatic feature learning in RL for large or continuous state spaces.
method Bootstrapping methods and theoretical analysis of temporal difference learning.
result State representations differ from other auxiliary-task-based approaches.
KBB algorithm reduces sample complexity for policy evaluation in general state spaces.
problem Policy evaluation in large state spaces with high sample complexity.
method Alternates between fitting Bellman residual and estimating value function via adaptive feature set growth.
result Super-linear convergence rates demonstrated, with reductions in sample complexity.
RFC enhances humanoid control to imitate complex human motions.
problem Dynamics mismatch between humanoid models and real humans.
method Residual Force Control (RFC) augments control policies with external forces.
result RFC outperforms state-of-the-art methods in convergence speed and motion quality.
Proposes log density gradient to improve reinforcement learning sample complexity.
problem Residual error in gradient estimation in policy gradient methods.
method Log density gradient method to correct residual error, using state-action discounted distributional formulation.
result Min-max optimization method to approximate log density gradient with on-policy samples, achieving sample complexity of m−1/2. Study tight offline learning bounds for linear MDPs using variance information.
problem Understanding statistical limits with linear function representations in offline reinforcement learning.
method Variance-aware pessimistic value iteration (VAPVI) that reweights Bellman residuals based on estimated variances.
result Improved offline learning bounds expressed in terms of system quantities.
New estimator improves off-policy evaluation for large action spaces.
problem Conventional importance-weighting approaches suffer from excessive variance in off-policy evaluation for large discrete action spaces.
method Proposes OffCEM estimator based on conjunct effect model (CEM), applying importance weighting only to action clusters and using model-based reward estimation for residual effects.
result Proposed estimator is unbiased under local correctness condition, providing substantial improvements in OPE especially with many actions.
Residual Networks are shown to be equivalent to boosting feature representation.
problem Improving feature representation in deep learning models.
method Proved ResNet's equivalence to Online Gradient Boosting and proposed decision tree residual modules.
result ResNet can achieve Online Gradient Boosting regret bounds through architectural changes.
Recent work in fairness in machine learning has proposed adjusting for fairness by equalizing accuracy metrics across groups and has also studied how datasets affected by historical prejudices may lead to unfair decision policies. We connect these lines of work and study the residual unfairness that arises when a fairn…
MULTIPOLAR aggregates diverse source policies for efficient transfer RL.
problem Efficiently transfer knowledge between different environmental dynamics.
method Adaptive action aggregation and residual prediction network.
result Demonstrated effectiveness across diverse simulated environments.
New method uses observational data to improve trial design efficiency.
problem Scarce randomized controlled trials; inefficiency of using observational data.
method Active Residual Learning, R-Design framework, R-EPIG criterion.
result Efficiently estimating residuals to correct observational bias improves trial design.
In this paper, we introduce transformations of deep rectifier networks, enabling the conversion of deep rectifier networks into shallow rectifier networks. We subsequently prove that any rectifier net of any depth can be represented by a maximum of a number of functions that can be realized by a shallow network with a …
CEFOL uses deep learning for dynamic programming with recursive utility.
problem Challenges in solving dynamic programming problems with recursive utility.
method Introduces a separate neural network for certainty equivalent, uses first-order optimality conditions to learn value and policy functions.
result CEFOL achieves high accuracy in learning value and policy functions, matching VFI benchmarks.
Spatial Adapter adds structured spatial representation to frozen predictors.
problem Efficiently adding spatial structure to pre-trained models.
method Structured spatial decomposition and closed-form covariance for residual fields.
result Adapter improves spatial prediction and uncertainty quantification.
Deep reinforcement learning, applied to vision-based problems like Atari games, maps pixels directly to actions; internally, the deep neural network bears the responsibility of both extracting useful information and making decisions based on it. By separating the image processing from decision-making, one could better …
Minimal assumptions analysis of Q-learning with time-varying policies.
problem Finite-time analysis of Q-learning with time-varying policies for discounted MDPs.
method Minimal assumptions, Poisson equation decomposition, sensitivity analysis.
result Established convergence rate and sample complexity for Q-learning.
New method improves deep policy gradient algorithms by learning relative state values.
problem High sample complexity and instability in policy gradient methods.
method Uses a new state-value function approximation based on residual variance.
result Empirical improvement across diverse continuous control tasks and algorithms.
New method for evaluating and learning in complex decision-making scenarios.
problem Evaluating and learning from policies in contextual combinatorial bandits with high bias and variance.
method Factored action space decomposition and importance sampling-based estimator (OPCB).
result OPCB achieves superior performance in OPE and OPL compared to conventional methods.
RFRBoost uses random features to boost deep residual neural networks, improving performance and computational efficiency.
problem Improving performance of deep residual neural networks (RFNNs) while preserving convex optimization benefits.
method Random Feature Representation Boosting (RFRBoost) using boosting theory and random features at each layer.
result RFRBoost significantly outperforms RFNNs and end-to-end trained MLP ResNets in small- to medium-scale tabular datasets.
A new model improves CT image quality from low-dose scans.
problem Improving CT image quality from low-dose scans.
method Multi-layer Residual Sparsifying Transform (MRST) learning model for low-dose CT reconstruction.
result The MRST model outperforms conventional methods in maintaining subtle details.
A method to reduce bias in model-based policy evaluation by shifting operators.
problem Bias in value function computation from noisy estimated models.
method Operator shifting method to reduce the residual norm error.
result The shifting factor is always positive and upper bounded by $1+O\left(1/n
ight)$.
Temporal difference learning and Residual Gradient methods are the most widely used temporal difference based learning algorithms; however, it has been shown that none of their objective functions is optimal w.r.t approximating the true value function V. Two novel algorithms are proposed to approximate the true value…
Develop a decision-calibrated conformal framework for pacing decisions in streaming advertising.
problem Pacing decisions in streaming advertising
method Develop a decision-calibrated conformal framework
result The proposed score is the smallest valid uncertainty measure that uniformly protects all deployable pacing policies.
Global analysis of Dixmier traces and Wodzicki residues on compact Lie groups.
problem Computing Dixmier traces and Wodzicki residues on compact Lie groups.
method Global quantisation approach, using global symbols and representation theory.
result Explicit formulae for Dixmier traces and Wodzicki residues on compact Lie groups.
RED-SC improves deep subspace clustering efficiency.
problem Slow convergence in training deep subspace clustering models.
method Residual Encoder-Decoder network with skip-layer connections and self-expressive layer.
result Training converges much faster with RED-SC.
SCORE improves tree-based predictions with boosted residual extraTrees.
problem Improving tree-based prediction models with reduced errors.
method Inspired by representation learning, SCORE uses boosting, regularized regression, and variable selection.
result SCORE provides comparable or superior performance compared to other models.
POLAR learns efficient data acquisition policies using pretrained belief representations.
problem Challenges in learning effective policies for adaptive data acquisition.
method POLAR decouples representation learning from policy learning by leveraging pretrained predictive foundation models as belief-state encoders.
result POLAR outperforms state-of-the-art methods across diverse tasks while requiring fewer training samples.
ScoreMatchingRiesz improves debiased machine learning and policy effects estimation.
problem Improving debiased machine learning and policy effects estimation.
method Score matching and Riesz representer estimation.
result Estimates policy path for continuous treatments, improving interpretability.
VA-OPE improves OPE by incorporating variance information, achieving tighter error bounds.
problem Estimating value function of a target policy from offline data collected by a behavior policy.
method Proposes VA-OPE, an algorithm that reweights Bellman residual using estimated variance of the value function.
result Achieves a tighter error bound than the best-known result.
GResNet tackles suspended animation in deep GNNs by adding extensive connections.
problem Performance degradation of deep GNNs, especially spectral-based models.
method Introducing GResNet framework with extensively connected highways.
result Avoids dramatic changes to node representations between layers, enhancing learnability.
Study the intersection of positive closed currents using tangent currents and King's residue formula.
problem Investigate the intersection of positive closed currents in complex manifolds.
method Employ tangent currents and King's residue formula to establish a natural condition for intersection.
result Derive an integral representation of the intersection of positive closed currents.
BCRL learns a Bellman complete representation for offline RL policy evaluation.
problem Learning a Q-function efficiently from offline data.
method BCRL learns a linear Bellman complete representation directly from data, enabling efficient OPE.
result BCRL achieves competitive OPE error and outperforms FQE in certain scenarios.
Study of resonances and residue operators for hyperbolic spaces.
problem Understanding resonances and residue operators for pseudo-Riemannian hyperbolic spaces.
method Analyzing the resolvent of the Laplace-Beltrami operator on pseudo-Riemannian hyperbolic spaces.
result Explicit determination of resonances and identification of residue representations.
Invertible DenseNets improve model efficiency and performance.
problem Improving model efficiency and performance in neural networks.
method Enforcing invertibility in DenseNets by satisfying the Lipschitz constraint and proposing a learnable concatenation.
result i-DenseNets outperform Residual Flows in negative log-likelihood on various datasets.
Recurrent neural networks (RNNs) are an effective representation of control policies for a wide range of reinforcement and imitation learning problems. RNN policies, however, are particularly difficult to explain, understand, and analyze due to their use of continuous-valued memory vectors and observation features. In …
This work explores representation complexity in RL paradigms, revealing model-based RL as the easiest task.
problem Investigating the representation complexity gap among model-based, policy-based, and value-based RL.
method Demonstrated through analysis of Markov decision processes (MDPs) and introduced new classes of MDPs.
result Representation complexity hierarchy: model-based RL > policy-based RL > value-based RL.
New model explains how concepts grow based on experience.
problem Existing models assume fixed representation; new model allows for growth.
method Geometric framework with MDL criterion for basis extension.
result Conceptual growth is selective and conservative, exposing or amplifying residual error.
The paper shows how to stabilize off-policy reinforcement learning using specific state representations.
problem Stability issues in reinforcement learning with function approximation and off-policy learning.
method Formal analysis of representation learning schemes based on the transition matrix of a policy.
result Schur and orthogonal bases of the Krylov subspace provide stable representations for TD learning.
We identify action representations from video data, proving their statistical benefits.
problem Identifying latent action policies from video data.
method Entropy-regularized LAPO objective, formalizing desiderata for action representations.
result Entropy-regularized LAPO identifies action representations satisfying desiderata under suitable conditions.
There is a growing interest in learning data representations that work well for many different types of problems and data. In this paper, we look in particular at the task of learning a single visual representation that can be successfully utilized in the analysis of very different types of images, from dog breeds to s…
Probabilistic principal component analysis (PPCA) seeks a low dimensional representation of a data set in the presence of independent spherical Gaussian noise, Sigma = (sigma^2)*I. The maximum likelihood solution for the model is an eigenvalue problem on the sample covariance matrix. In this paper we consider the situa…
DOLCE improves off-policy evaluation and learning by decomposing effects.
problem Bias in off-policy evaluation and learning due to policy mismatch.
method Uses lagged contexts and a moment-based training procedure to decompose and cancel bias.
result DOLCE achieves substantial improvements in off-policy evaluation and learning.
We present a new approach to the problems of evaluating and learning personalized decision policies from observational data of past contexts, decisions, and outcomes. Only the outcome of the enacted decision is available and the historical policy is unknown. These problems arise in personalized medicine using electroni…
We study the problem of off-policy policy evaluation (OPPE) in RL. In contrast to prior work, we consider how to estimate both the individual policy value and average policy value accurately. We draw inspiration from recent work in causal reasoning, and propose a new finite sample generalization error bound for value e…