A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Paper explores weighted averaging schemes for SGD, achieving asymptotic normality and optimality.
New class of heavy-tailed distributions shows weighted averages dominate individual variables.
Improved averaging method for noisy observations converges strongly.
We propose and analyze a variant of the classic Polyak-Ruppert averaging scheme, broadly used in stochastic gradient methods. Rather than a uniform average of the iterates, we consider a weighted average, with weights decaying in a geometric fashion. In the context of linear least squares regression, we show that this …
We propose Stochastic Weight Averaging in Parallel (SWAP), an algorithm to accelerate DNN training. Our algorithm uses large mini-batches to compute an approximate solution quickly and then refines it by averaging the weights of multiple models computed independently and in parallel. The resulting models generalize equ…
New averaging technique speeds up Newton method convergence.
Dropout and similar stochastic neural network regularization methods are often interpreted as implicitly averaging over a large ensemble of models. We propose STE (stochastically trained ensemble) layers, which enhance the averaging properties of such methods by training an ensemble of weight matrices with stochastic r…
We propose methods for distributed graph-based multi-task learning that are based on weighted averaging of messages from other machines. Uniform averaging or diminishing stepsize in these methods would yield consensus (single task) learning. We show how simply skewing the averaging weights or controlling the stepsize a…
In this note, we present a new averaging technique for the projected stochastic subgradient method. By using a weighted average with a weight of t+1 for each iterate w_t at iteration t, we obtain the convergence rate of O(1/t) with both an easy proof and an easy implementation. The new scheme is compared empirically to…
Unified analysis of finite weight averaging methods in deep learning.
The study introduces anytime learning schedules for large language models without fixed horizons.
Designing a deep neural network (DNN) with good generalization capability is a complex process especially when the weights are severely quantized. Model averaging is a promising approach for achieving the good generalization capability of DNNs, especially when the loss surface for training contains many sharp minima. W…
This paper analyzes SGD with increasingly weighted averaging for optimization and generalization.
Improved neural networks by averaging late-stage weights.
Proposes SWA for adversarial training to improve model robustness.
Unexpectedly, weighted Pareto variables are stochastically dominant.
In this paper, we focus on quantifying model stability as a function of random seed by investigating the effects of the induced randomness on model performance and the robustness of the model in general. We specifically perform a controlled study on the effect of random seeds on the behaviour of attention, gradient-bas…
Consider a family of portfolio strategies with the aim of achieving the asymptotic growth rate of the best one. The idea behind Cover's universal portfolio is to build a wealth-weighted average which can be viewed as a buy-and-hold portfolio of portfolios. When an optimal portfolio exists, the wealth-weighted average c…
Algorithm learns stochastic system dynamics from data.
We formulate and study a general family of (continuous-time) stochastic dynamics for accelerated first-order minimization of smooth convex functions. Building on an averaging formulation of accelerated mirror descent, we propose a stochastic variant in which the gradient is contaminated by noise, and study the resultin…
Two-Tailed Averaging improves generalization by optimizing the number of leading iterates to ignore.
A new accelerated method with simpler momentum update rules.
We propose an explicit recursive method to approximate a power-law with a finite sum of weighted exponentials. Applications to moving averages with long memory are discussed in relationship with stochastic volatility models.
Simplified analysis of SGD for linear regression with weight averaging.
New study shows diversification can increase risk for heavy-tailed losses.
Conventional Monte Carlo simulations are stochastic in the sense that the acceptance of a trial move is decided by comparing a computed acceptance probability with a random number, uniformly distributed between 0 and 1. Here we consider the case that the weight determining the acceptance probability itself is fluctuati…
Improving optimization for iterate-averaged language models
Presently the most successful approaches to semi-supervised learning are based on consistency regularization, whereby a model is trained to be robust to small perturbations of its inputs and parameters. To understand consistency regularization, we conceptually explore how loss geometry interacts with training procedure…
Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple averaging of multiple points along the trajectory of SGD, with a cyclical or constant learning rate, leads to better generalization than conv…
New method for online statistical inference in contextual bandits using SGD.
GACTGAN synthesizes tabular data better with less computational overhead.
We propose SWA-Gaussian (SWAG), a simple, scalable, and general purpose approach for uncertainty representation and calibration in deep learning. Stochastic Weight Averaging (SWA), which computes the first moment of stochastic gradient descent (SGD) iterates with a modified learning rate schedule, has recently been sho…
Bayesian method improves adaptive testing item selection, ensuring full item exposure.
BEMA reduces bias in EMA, leading to faster convergence and better performance.
New bounds for SGD show improved performance in various settings.
PACE optimizes training for averaged language models, improving performance.
We give a new proof of the representation of implied volatility as a time-average of weighted expectations of local or stochastic volatility. With this proof we clarify the question of existence of 'forward implied variance' in the original derivation of Gatheral, who introduced this representation in his book 'The Vol…
Low precision operations can provide scalability, memory savings, portability, and energy efficiency. This paper proposes SWALP, an approach to low precision training that averages low-precision SGD iterates with a modified learning rate schedule. SWALP is easy to implement and can match the performance of full-precisi…
In this short note, we study an optimization problem of expected implementation shortfall (IS) cost under general shaped market impact functions. In particular, we find that an optimal strategy is a VWAP (volume weighted average price) execution strategy when the market model is a Black-Scholes type with stochastic clo…
A new method calibrates scientific models by adding randomness to their predictions.
A new algorithm flattens multi-modal distributions for better deep learning.
Enhances linear regression with Kalman filter for loss minimization.
Investment strategy optimizes risk using a specific risk measure.
WASH trains ensembles with shuffled weights to improve accuracy and reduce communication.
New algorithm samples Bayesian neural networks for improved calibration.
Multi-layer neural networks have lead to remarkable performance on many kinds of benchmark tasks in text, speech and image processing. Nonlinear parameter estimation in hierarchical models is known to be subject to overfitting and misspecification. One approach to these estimation and related problems (local minima, co…
Paper uses Mirror Descent for efficient risk budgeting portfolios.
New moving average adapts weight dynamically based on polynomial and wavefunction.