Improving optimization for iterate-averaged language models
problem How to optimize the averaged model returned by Language Model pipelines
method Formulating optimizer design as an optimal-control problem
result Proven convergence rate and strict improvement in squared error
The paper tackles learning to control systems with unknown parameters using Brownian noise.
problem Learning to control systems with unknown parameters.
method Proposes algorithms based on moving empirical averages and integrates statistical methods with stochastic control theory.
result Achieves a logarithmic expected regret rate.
PACE optimizes training for averaged language models, improving performance.
problem How to optimize training for averaged language model iterates.
method Formulated as an optimal-control problem, solved for minimizing error of the average with a penalty on intervention size.
result PACE improves the limiting squared error of the iterate-average estimator by an arbitrarily large factor on some instances.
One of the fundamental assumptions in stochastic control of continuous time processes is that the dynamics of the underlying (diffusion) process is known. This is, however, usually obviously not fulfilled in practice. On the other hand, over the last decades, a rich theory for nonparametric estimation of the drift (and…
New method approximates controllability of large networks from coarse summaries.
problem Controlling large-scale linear dynamical systems with incomplete network information.
method Algorithm using stochastic block model to estimate controllability from coarse summaries.
result Average controllability of fine-scale system can be well approximated by coarse-scale system.
Improved averaging method for noisy observations converges strongly.
problem Noisy observations from random dynamical systems require stable estimates.
method Introduced p-EMA, a modified exponential moving average with subharmonic weight decay. result Stochastic convergence guarantees for p-EMA under mild assumptions. We propose methods for distributed graph-based multi-task learning that are based on weighted averaging of messages from other machines. Uniform averaging or diminishing stepsize in these methods would yield consensus (single task) learning. We show how simply skewing the averaging weights or controlling the stepsize a…
Optimizes control of noisy discrete systems without system matrix knowledge.
problem Optimal control of discrete-time systems with additive and multiplicative noises.
method Stochastic Lyapunov and Riccati equations, model-free reinforcement learning.
result Model-free reinforcement learning algorithm converges to optimal control policy.
Paper proposes SCALLION and SCAFCOM for compressed FL with reduced communication.
problem Reducing communication overhead in Federated Learning with data heterogeneity and partial participation.
method Revisit and simplify stochastic controlled averaging, proposing SCALLION and SCAFCOM for unbiased and biased compression.
result SCALLION and SCAFCOM outperform existing methods in communication and computation complexities.
Optimal withdrawal strategy for DC pension plans maximizes total withdrawals while managing risk.
problem Maximizing withdrawals from DC pension plans while managing risk.
method Optimal stochastic control approach with constraints on withdrawal and asset allocation.
result Optimal strategy yields higher average withdrawals with minimal increase in risk.
Optimized certainty equivalents (OCEs) is a family of risk measures widely used by both practitioners and academics. This is mostly due to its tractability and the fact that it encompasses important examples, including entropic risk measures and average value at risk. In this work we consider stochastic optimal control…
This work proposes an online learning approach to tighten constraints in stochastic control problems.
problem Solving chance-constrained stochastic optimal control problems is computationally challenging.
method Reformulate chance constraints as a binary regression problem and use a GP model to learn constraint-tightening parameters online.
result The approach tightens constraints more effectively, leading to lower costs in numerical experiments.
In this paper, we focus on quantifying model stability as a function of random seed by investigating the effects of the induced randomness on model performance and the robustness of the model in general. We specifically perform a controlled study on the effect of random seeds on the behaviour of attention, gradient-bas…
Study of LQ MFGs in infinite-dimensional Hilbert spaces.
problem Mean field games in infinite-dimensional settings with stochastic dynamics.
method Analysis of coupled semilinear infinite-dimensional stochastic evolution equations, development of Nash equilibrium.
result Characterization of unique Nash equilibrium in the limit of many agents.
Derives time-averaged active inference from control principles.
problem Finite-horizon or discounted-surprise problems in active inference.
method Derives infinite-horizon, average-surprise active inference from optimal control principles.
result Unified objective functional for sensorimotor control.
Proposes a new resampling method for off-policy evaluation in stochastic control.
problem Estimating policy performance from data generated under a different policy.
method K-nearest neighbor resampling procedure for off-policy evaluation.
result Statistical consistency results for the proposed method under weak conditions.
A new sampling method estimates scores without training or nested MCMC.
problem Efficient sampling from complex, unnormalised distributions.
method Multiscale averaging in SDEs for score estimation.
result Empirical results show competitive accuracy and efficiency.
A new method calibrates scientific models by adding randomness to their predictions.
problem Current scientific foundation models lack calibrated uncertainty.
method Stochastic Attention, which randomizes attention weights using multinomial samples.
result Stochastic Attention achieves the strongest native calibration and sharpest prediction intervals.
New algorithm optimizes stochastic optimization with circular dependency.
problem Circular dependency between decision variable and importance sampling.
method Single-loop stochastic approximation algorithm based on Nesterov's dual averaging.
result Achieves minimal asymptotic variance and resolves circular optimization challenge.
A new method reduces variance in SGMCMC by preferentially subsampling data.
problem High variance in stochastic gradient estimates impacts sampler performance.
method Use a non-uniform probability distribution to preferentially subsample data points and adaptively adjust subsample size.
result Maintains accuracy while substantially reducing average subsample size.
We seek a discussion about the most suitable feedback control structure for stock trading under the consideration of proportional transaction costs. Suitability refers to robustness and performance capability. Both are tested by considering different one-step ahead prediction qualities, including the ideal case, correc…
Federated Averaging (FedAvg) has emerged as the algorithm of choice for federated learning due to its simplicity and low communication cost. However, in spite of recent research efforts, its performance is not fully understood. We obtain tight convergence rates for FedAvg and prove that it suffers from `client-drift' w…
Deep RL improves power control and scheduling for wireless multicast systems.
problem Scalable power control and scheduling for wireless multicast networks.
method Deep reinforcement learning with function approximation using a deep neural network.
result Deep RL can learn optimal power control policies for large systems.
BCI provides calibrated prediction intervals for time series forecasts.
problem Calibration of prediction intervals for time series forecasts.
method BCI wraps around any time series forecasting models and optimizes interval lengths using dynamic programming.
result BCI achieves long-term coverage under arbitrary distribution shifts and temporal dependence.
Study of multidimensional control problems with reflection controls.
problem Solving control problems with reflection controls in multidimensional settings.
method Gradient descent algorithm for polytope approximations, data-driven domain estimator, episodic learning algorithm.
result Data-driven solutions for unknown diffusion dynamics with sublinear regret.
Improved stability for large-scale Bayesian sampling.
problem Reducing instability in Langevin dynamics for large datasets.
method Introducing a modified CCAdL thermostat with a scaling and squaring method and a truncated Taylor series approximation.
result Significantly improved numerical stability and accuracy over existing methods.
New method reduces sample complexity for robust reinforcement learning.
problem Finite sample analysis in robust reinforcement learning.
method Stochastic approximation framework with controlled bias, using MLMC techniques and geometric truncation.
result Order-optimal sample complexity of ildeO(ε−2) for robust policy evaluation. Deviation inequalities for stochastic approximation methods.
problem Establishing bounds on the deviation of stochastic approximation methods.
method Martingale approximation method for separately Lipschitz functions.
result Established various deviation inequalities for stochastic approximation by averaging and minimization.
High-dimensional models can outperform simpler ones in causal inference.
problem Estimating average treatment effects with many covariates.
method High-dimensional linear regression and synthetic control with many control units.
result Adding more control units can improve imputation performance even when pre-treatment fit is perfect.
New theory for nonsmooth systems helps optimize and control complex functions.
problem Optimizing and controlling systems with nonsmooth functions.
method Higher-order averaging theory with nonsmooth near-identity transformation and lexicographic differentiation.
result Closed formula for nonsmooth first and second-order averaging.
Investment strategy optimizes risk using a specific risk measure.
problem Optimizing investment with risk controlled by a weighted entropic risk measure.
method Investigation of expected utility maximization and risk minimization problems with solutions provided iteratively.
result Explicit characterization of solutions to optimization problems.
RL and DTSOC for final quadratic hedging performance studied.
problem Optimal hedging of European call options with and without transaction costs.
method Reinforcement Learning and Deep Trajectory-based Stochastic Optimal Control.
result RL and DTSOC perform similarly to variance-optimal hedging in various market models.
We propose and analyze a variant of the classic Polyak-Ruppert averaging scheme, broadly used in stochastic gradient methods. Rather than a uniform average of the iterates, we consider a weighted average, with weights decaying in a geometric fashion. In the context of linear least squares regression, we show that this …
The paper considers a class of multi-agent Markov decision processes (MDPs), in which the network agents respond differently (as manifested by the instantaneous one-stage random costs) to a global controlled state and the control actions of a remote controller. The paper investigates a distributed reinforcement learnin…
Paper develops bounds for stochastic approximation with averaging.
problem Establish high-probability bounds for averaged stochastic approximation.
method Develops a general framework for non-asymptotic concentration bounds.
result Derives sharp bounds for averaged iterates and tightens existing results.
This paper tackles robust control of noisy systems with uncertain distributions.
problem Optimal control of sampled-data stochastic systems with multiplicative noise and distributional ambiguity.
method Develops a convex relaxation to handle the ``concave-max'' geometry and derives a probabilistic performance guarantee.
result Derives an explicit, non-asymptotic bound on the duality gap and proves robust viability conditions.
Paper proposes faster method to find local minima in nonconvex optimization.
problem Escaping saddle points and finding local minima in nonconvex optimization.
method LENA (Last stEp shriNkAge) framework for faster perturbed stochastic gradient methods.
result LENA finds (ε,εH)-approximate local minima within ildeO(ε−3+εH−6) evaluations. Paper explores weighted averaging schemes for SGD, achieving asymptotic normality and optimality.
problem Improving convergence of SGD in various settings.
method Develops a general weighted averaging scheme for SGD and establishes asymptotic normality.
result Establishes asymptotic normality and optimality of weighted averaged SGD solutions.
Dropout and similar stochastic neural network regularization methods are often interpreted as implicitly averaging over a large ensemble of models. We propose STE (stochastically trained ensemble) layers, which enhance the averaging properties of such methods by training an ensemble of weight matrices with stochastic r…
New method assesses financial and cyber risks under uncertainty.
problem Uncertainty in risk assessment for financial and cyber systems.
method Combines stochastic approximation and distorted mix method to compute worst case average value at risk.
result Efficient algorithm for tail uncertainty in multivariate distributions.
This paper augments the reward received by a reinforcement learning agent with potential functions in order to help the agent learn (possibly stochastic) optimal policies. We show that a potential-based reward shaping scheme is able to preserve optimality of stochastic policies, and demonstrate that the ability of an a…
New averaging technique speeds up Newton method convergence.
problem Superlinear convergence of stochastic Newton methods with noisy Hessians.
method Hessian averaging to reduce noise and maintain superlinear convergence.
result Hessian averaging achieves superlinear convergence with a non-asymptotic rate.
Investigates portfolio selection among competitive agents with mean-variance preferences.
problem Optimizing portfolios with multi-agent competition and relative wealth comparison.
method Reformulated as a constrained, non-homogeneous stochastic linear-quadratic control problem; derived optimal feedback strategies; used decoupling techniques and fixed-point theory to solve nonlinear BSDEs.
result Characterized three scenarios based on market and competition parameters: unique Nash equilibrium, no Nash equilibrium, or infinitely many Nash equilibria.
Paper studies central bank's strategy to control systemic risk in interbank system.
problem Minimizing average distance between log-monetary reserves and target levels.
method Weak formulation, Ekeland's variational principle, Gamma-convergence, stochastic Fokker-Planck-Kolmogorov equation.
result Proves convergence of optimal strategies as number of banks increases.
New class of heavy-tailed distributions shows weighted averages dominate individual variables.
problem Understanding and comparing risks in heavy-tailed distributions.
method Introducing a new class of heavy-tailed distributions and proving stochastic dominance relations.
result Weighted averages of random variables in this class are stochastically larger than individual variables.
Optimizes control of infectious disease spread using stochastic methods.
problem Optimizing control of highly infectious diseases like COVID-19.
method Reformulated Hamilton-Jacobi-Bellman equation as stochastic minimum principle, leading to forward-backward stochastic differential equations.
result Numerous numerical solutions presented under various scenarios.
Paper solves a control problem with robust methods.
problem Monotone mean-variance problems with stochastic coefficients.
method Finding saddle point through BSDEs with unbounded coefficients.
result Optimal control and value match mean-variance problems.
Study the averaging principle for non-autonomous slow-fast systems and apply it to financial local stochastic volatility models.
problem Understanding the behavior of non-autonomous slow-fast systems of stochastic differential equations.
method Prove the averaging principle under specific conditions and apply it to a financial model.
result Prices of derivatives converge to those calculated using the limit model under a risk-neutral measure.