Softmax Bellman operator improves Q-function performance in RL despite sub-optimality.
problem Softmax Bellman operator's impact on value functions in RL is problematic.
method Revisited theoretical properties of softmax Bellman operator, proving convergence and overestimation reduction.
result Softmax Bellman operator leads to superior policies in practice, even outperforming double Q-learning.
New method improves stability of soft FQI for offline RL.
problem Stability issues in soft FQI under function approximation.
method Stationary reweighting to align operator norms.
result Local linear convergence proved under certain conditions.
Linear Q-learning converges to a bounded set without divergence.
problem Proving linear Q-learning does not diverge and converges to a bounded set.
method No modifications to the original linear Q-learning algorithm, no Bellman completeness or near-optimality assumptions, only an ε-softmax behavior policy with adaptive temperature.
result First L2 convergence rate of linear Q-learning iterates to a bounded set. A softmax operator applied to a set of values acts somewhat like the maximization function and somewhat like an average. In sequential decision making, softmax is often used in settings where it is necessary to maximize utility but also to hedge against problems that arise from putting all of one's weight behind a sing…
DBS improves convergence in reinforcement learning.
problem Softmax operator convergence issues in reinforcement learning.
method Dynamic Boltzmann Softmax (DBS) updates value function.
result DBS enables better value function estimation and convergence.
Softmax is found ineffective for NL block, leading to improved performance.
problem Inefficiency of softmax in NL block for global context modeling.
method Empirical analysis and replacement of softmax with scaling factor.
result Improved performance on various datasets with reduced computational cost.
Study solves optimal portfolio selection using HJB equation.
problem Optimal portfolio selection problem.
method Maximal monotone operator method, Banach fixed-point theorem, Fourier transform, monotone operators technique.
result Existence and uniqueness of solution to HJB equation.
Unified framework for studying softmax attention under large prompts.
problem Challenges in theoretical analysis of softmax attention.
method Measure-based framework for finite and infinite prompts.
result Softmax attention converges to linear attention in the large-prompt regime.
A new operator based on t-distributions improves NN classifiers' robustness to out-of-distribution samples.
problem NN classifiers assign extreme probabilities to out-of-distribution samples, leading to unreliable predictions.
method Derive a novel operator using t-distributions to model uncertainty more accurately.
result Classifiers using the new operator are more robust to out-of-distribution samples.
DSPI connects natural policy gradient to policy iteration, proving global convergence.
problem Optimizing policies in reinforcement learning.
method DSPI framework, combining smoothed policy iteration and natural policy gradient.
result DSPI achieves geometric convergence and optimal complexity for policy optimization.
The paper analyzes off-policy TD-learning using generalized Bellman operators and provides finite-sample bounds.
problem High variance in off-policy TD-learning due to importance sampling.
method Derives finite-sample bounds for off-policy TD-like algorithms using generalized Bellman operators.
result First-known finite-sample guarantees for several off-policy TD algorithms.
Richard Bellman's Principle of Optimality, formulated in 1957, is the heart of dynamic programming, the mathematical discipline which studies the optimal solution of multi-period decision problems. In this paper, we look at the main trading principles of Jesse Livermore, the legendary stock operator whose method was pu…
In this paper, we consider the stochastic iterative counterpart of the value iteration scheme wherein only noisy and possibly biased approximations of the Bellman operator are available. We call this counterpart as the approximate value iteration (AVI) scheme. Neural networks are often used as function approximators, i…
New algorithms reduce complexity for learning in MDPs with entropy regularization.
problem Efficient learning for MDPs with large or continuous state and action spaces.
method Multilevel Monte Carlo (MLMC) algorithms integrating fixed-point iteration and stochastic approximation of the Bellman operator.
result MLMC with unbiased approximation of the Bellman operator achieves polynomial sample complexity.
New method stabilizes FQE by reweighting Bellman targets.
problem Stability guarantees for FQE often rely on Bellman completeness, which can fail with function approximation.
method Proposes stationary-weighted FQE, reweighting Bellman targets by stationary target-to-behavior density ratio.
result Proves finite-sample linear convergence to stationary projected Bellman fixed point without Bellman completeness.
A new estimator combines bootstrapping and rollout methods in RL.
problem Combining strengths of bootstrapping and rollout methods in RL.
method Subgraph Bellman operators and fixed point solving.
result Upper bound on error approaches optimal TD variance with additional term.
One-step Bellman alignment improves online RL by reducing task mismatch.
problem Online RL struggles with task similarity defined by rewards or transitions.
method One-step Bellman alignment and re-weighted targeting (RWT) to correct task mismatch.
result Regret bounds show task shift complexity, not target MDP, affects performance.
ZeroS improves Transformers by adding negative weights, matching or beating softmax attention.
problem Limited performance of linear attention methods, especially in long context sequences.
method Proposes Zero-Sum Linear Attention (ZeroS) that removes the zero-order term and reweights zero-sum softmax residuals.
result ZeroS matches or exceeds standard softmax attention across various benchmarks, theoretically expanding representable functions.
FORE evaluates occupancy ratios without requiring Bellman completeness.
problem Offline reinforcement learning occupancy ratio estimation.
method Fitted occupancy-ratio evaluation (FORE) using adjoint Bellman recursion.
result FORE achieves convergence in KL without Bellman completeness.
The paper tackles non-cumulative objectives in reinforcement learning and proposes modifications to existing algorithms.
problem Optimizing objectives that are not naturally expressed as summations of rewards in various fields.
method The paper modifies the Bellman optimality equation to handle non-cumulative objectives by replacing summation with a generalized operation.
result The modified Bellman updates can converge to the globally optimal solution under certain conditions.
Paper proposes Sinkformers for Transformers with doubly stochastic attention.
problem Improving Transformer models' accuracy in vision and natural language processing.
method Using Sinkhorn's algorithm to make attention matrices doubly stochastic instead of SoftMax normalization.
result Sinkformers enhance model accuracy in vision and natural language processing tasks.
The paper tackles data-driven optimal control of unknown nonlinear systems using RKHS.
problem Unknown nonlinear dynamics and stage cost functions.
method Embed state densities into RKHS, learn Markov operators, solve Hamilton-Jacobi-Bellman recursions.
result Solves a wide range of nonlinear control problems, including depth regulation.
The paper solves a complex financial optimization problem using a novel mathematical technique.
problem Optimizing portfolio selection in financial markets.
method Maximal monotone operator method and Riccati transformation.
result Existence and uniqueness of a solution to the transformed parabolic equation in a Sobolev space.
Market makers optimize trading with a new implicit scheme for complex inequalities.
problem Optimizing trading in a limit order book with stochastic and impulse control.
method Implicit numerical scheme coupled with policy iteration algorithm.
result Convergence to the unique viscosity solution of the HJBQVI.
New method learns policies from offline data using operator models.
problem Limited understanding of approximation errors in offline reinforcement learning.
method Linking reinforcement learning to Hamilton-Jacobi-Bellman equation, proposing operator-theoretic algorithm.
result Global convergence of the value function and finite-sample guarantees derived.
We consider a new family of operators for reinforcement learning with the goal of alleviating the negative effects and becoming more robust to approximation or estimation errors. Various theoretical results are established, which include showing on a sample path basis that our family of operators preserve optimality an…
New non-linear models for better preference predictions.
problem Improving models for preference orderings.
method Developed a general class of non-linear Bellman equations.
result New models can make different predictions under various circumstances.
Efficient algorithm for evaluating hierarchical classification methods at multiple operating points.
problem Evaluating hierarchical classification methods at multiple operating points.
method Efficient algorithm to produce operating characteristic curves for any method that assigns scores to every class in the hierarchy.
result Top-down classifiers are dominated by a naive flat softmax classifier across the entire operating range.
Stochastic differential equation approximation for linear TD(0) under Markovian noise
problem Temporal-difference learning with linear function approximation
method Stochastic differential equation approximation
result Explains the constant-stepsize error floor
Paper establishes DRL for high-dimensional rewards.
problem Intractable reinforcement learning with high-dimensional rewards.
method Theoretical foundations and a novel DRL algorithm.
result Bellman operator contraction in high-dimensional spaces.
Permutations and matchings are core building blocks in a variety of latent variable models, as they allow us to align, canonicalize, and sort data. Learning in such models is difficult, however, because exact marginalization over these combinatorial objects is intractable. In response, this paper introduces a collectio…
Develops RF-softmax for faster training with softmax cross entropy.
problem High computational cost of training with softmax cross entropy.
method Random Fourier Features for efficient sampling from approximate softmax distribution.
result RF-softmax provides low bias in estimating both softmax distribution and its gradient.
Novel framework proves fast RL convergence in continuous spaces.
problem Analyzing stability in continuous state-action RL.
method Introduces a novel framework to analyze stability properties of RL.
result Highlights two key stability properties and demonstrates their satisfaction in RL.
Softmax is an output activation function for modeling categorical probability distributions in many applications of deep learning. However, a recent study revealed that softmax can be a bottleneck of representational capacity of neural networks in language modeling (the softmax bottleneck). In this paper, we propose an…
We study the tradeoff between computational effort and classification accuracy in a cascade of deep neural networks. During inference, the user sets the acceptable accuracy degradation which then automatically determines confidence thresholds for the intermediate classifiers. As soon as the confidence threshold is met,…
New RFs reduce kernel approximation variance and improve Transformer performance.
problem Efficient approximation of Gaussian and softmax kernels for kernel methods and Transformers.
method Parameterized, positive, non-trigonometric RFs optimized for variance reduction.
result Significant variance reduction in practice, outperforming previous methods.
New RL formulation for maximizing maximum reward in molecule generation.
problem Traditional RL frameworks do not fit real-world applications like drug discovery.
method Formulated a new objective function to maximize maximum reward, derived Bellman equation, introduced operators, and proved convergence.
result Achieved state-of-the-art results in molecule generation.
Survey of mathematical foundations for reinforcement learning.
problem Design and analysis of modern reinforcement learning algorithms.
method Organizes mathematical structures from probability, optimization, and operator theory.
result Unified mathematical entry point for researchers in various fields.
LARF improves random forests with attention mechanisms and contamination models.
problem Improving accuracy in classification tasks with random forests.
method Introduces a two-level attention mechanism and uses a mixture of contamination models.
result Significantly improved classification performance on various datasets.
This work is motivated by numerical solutions to Hamilton-Jacobi-Bellman quasi-variational inequalities (HJBQVIs) associated with combined stochastic and impulse control problems. In particular, we consider (i) direct control, (ii) penalized, and (iii) semi-Lagrangian discretization schemes applied to the HJBQVI proble…
Hierarchical Softmax approximates class probabilities for large datasets efficiently.
problem Computational inefficiency of Softmax for large-scale classification tasks.
method Used Hierarchical Softmax to approximate class probabilities efficiently.
result Hierarchical Softmax performance degrades as the number of classes increases.
Revises logistic-softmax likelihood for Bayesian meta-learning in few-shot classification.
problem Inherent uncertainty in logistic-softmax leads to suboptimal performance in meta-learning.
method Redesigns logistic-softmax likelihood with a temperature parameter for better control of prior confidence.
result Achieves well-calibrated uncertainty estimates and comparable/superior performance on benchmark datasets.
A new loss function improves reinforcement learning convergence.
problem Value function learning algorithms often fail to converge reliably.
method Proposes a novel kernel loss function for Bellman equation.
result The proposed loss function avoids divergence and works reliably.
New insights into CE dynamics reveal how Hadamard initialization simplifies softmax.
problem Understanding the dynamics of cross-entropy training loss in deep learning.
method Analyzing a two-layer linear neural network with standard-basis vectors as inputs.
result Gradient flow on cross-entropy converges to neural collapse geometry, proving global convergence.
Paper introduces Balanced Meta-Softmax for better long-tailed visual recognition.
problem Long-tailed distribution mismatch between training and testing data.
method Balanced Meta-Softmax, an unbiased extension of Softmax, using a Meta Sampler.
result Balanced Meta-Softmax outperforms state-of-the-art solutions on visual recognition and instance segmentation.
Paper introduces hierarchical softmax for global hierarchical classification tasks.
problem Improving classification accuracy in tasks with class hierarchies.
method Global hierarchical neural networks using hierarchical softmax.
result Hierarchical softmax outperforms regular softmax in multiple datasets.
DS-Softmax speeds up softmax inference by learning sparse experts.
problem Expensive softmax computations for large output classes.
method Sparse mixture of sparse experts for efficient top-k class retrieval.
result Significant computation reductions achieved at no performance loss.
Softmax emerges naturally in neural networks as a measure of conditional mutual information.
problem The artificial nature of softmax in neural networks.
method Information-theoretic perspective to derive log-softmax and evaluate conditional mutual information.
result Training deterministic neural networks through log-softmax maximises conditional mutual information.