A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Deviation inequalities for stochastic approximation methods.
We propose and analyze a variant of the classic Polyak-Ruppert averaging scheme, broadly used in stochastic gradient methods. Rather than a uniform average of the iterates, we consider a weighted average, with weights decaying in a geometric fashion. In the context of linear least squares regression, we show that this …
Improved averaging method for noisy observations converges strongly.
Paper develops bounds for stochastic approximation with averaging.
Paper explores weighted averaging schemes for SGD, achieving asymptotic normality and optimality.
Dropout and similar stochastic neural network regularization methods are often interpreted as implicitly averaging over a large ensemble of models. We propose STE (stochastically trained ensemble) layers, which enhance the averaging properties of such methods by training an ensemble of weight matrices with stochastic r…
New method assesses financial and cyber risks under uncertainty.
New averaging technique speeds up Newton method convergence.
New class of heavy-tailed distributions shows weighted averages dominate individual variables.
Study the averaging principle for non-autonomous slow-fast systems and apply it to financial local stochastic volatility models.
This work analyzes nonexpansive stochastic approximations with Markovian noise, proving convergence in reinforcement learning.
We consider a composite convex minimization problem associated with regularized empirical risk minimization, which often arises in machine learning. We propose two new stochastic gradient methods that are based on stochastic dual averaging method with variance reduction. Our methods generate a sparser solution than the…
New algorithms optimize spectral risk measures, improving interpolation between average and worst-case performance.
Optimal algorithms for Riemannian optimization with reduced complexity.
New method for unbiased regression reduces excess risk.
Improved SEG method converges to Nash equilibrium in bilinear games.
Study on stochastic approximation with Polyak-Ruppert averaging for linear systems.
Improving optimization for iterate-averaged language models
Averaged SGD achieves optimal convergence rate for neural networks in the NTK regime.
The paper develops time-uniform inference methods for stochastic approximation parameters.
The paper analyzes time-dependent streaming data with biased gradient estimates and proposes improved stochastic optimization methods.
We formulate and study a general family of (continuous-time) stochastic dynamics for accelerated first-order minimization of smooth convex functions. Building on an averaging formulation of accelerated mirror descent, we propose a stochastic variant in which the gradient is contaminated by noise, and study the resultin…
The purpose of this paper is to study the generalized Fong--Vasicek two-factor interest rate model with stochastic volatility. In this model the dispersion of the stochastic short rate (square of volatility) is assumed to be stochastic as well and it follows a non-negative process with volatility proportional to the sq…
Many machine learning, statistical inference, and portfolio optimization problems require minimization of a composition of expected value functions (CEVF). Of particular interest is the finite-sum versions of such compositional optimization problems (FS-CEVF). Compositional stochastic variance reduced gradient (C-SVRG)…
This work characterizes the benefits of averaging schemes widely used in conjunction with stochastic gradient descent (SGD). In particular, this work provides a sharp analysis of: (1) mini-batching, a method of averaging many samples of a stochastic gradient to both reduce the variance of the stochastic gradient estima…
New streaming methods improve convergence rates for optimization problems.
Paper uses averaging from many particle filters to approximate posterior predictive distributions.
We propose Stochastic Weight Averaging in Parallel (SWAP), an algorithm to accelerate DNN training. Our algorithm uses large mini-batches to compute an approximate solution quickly and then refines it by averaging the weights of multiple models computed independently and in parallel. The resulting models generalize equ…
We apply stochastic average gradient (SAG) algorithms for training conditional random fields (CRFs). We describe a practical implementation that uses structure in the CRF gradient to reduce the memory requirement of this linearly-convergent stochastic gradient method, propose a non-uniform sampling scheme that substant…
We briefly review our recent studies on stochastic processes modelling internet on-line trading. We present a way to evaluate the average waiting time between the observation of the price in financial markets and the next price change, especially in an on-line foreign exchange trading service for individual customers v…
Bayesian method improves adaptive testing item selection, ensuring full item exposure.
Averaged SGD optimizes a smoothed objective, leading to better generalization.
Stochastic gradient methods enable learning probabilistic models from large amounts of data. While large step-sizes (learning rates) have shown to be best for least-squares (e.g., Gaussian noise) once combined with parameter averaging, these are not leading to convergent algorithms in general. In this paper, we conside…
Paper approximates risk measures using SGD with Langevin dynamics.
Improved stochastic Halpern iteration for fixed-point approximation in normed spaces.
Improved stochastic optimization outperforms standard methods.
We consider stochastic gradient descent algorithms for minimizing a non-smooth, strongly-convex function. Several forms of this algorithm, including suffix averaging, are known to achieve the optimal convergence rate in expectation. We consider a simple, non-uniform averaging strategy of Lacoste-Julien et al. …
PACE optimizes training for averaged language models, improving performance.
In this note, we present a new averaging technique for the projected stochastic subgradient method. By using a weighted average with a weight of t+1 for each iterate w_t at iteration t, we obtain the convergence rate of O(1/t) with both an easy proof and an easy implementation. The new scheme is compared empirically to…
Stochastic Gradient Descent (SGD) is one of the simplest and most popular stochastic optimization methods. While it has already been theoretically studied for decades, the classical analysis usually required non-trivial smoothness assumptions, which do not apply to many modern applications of SGD with non-smooth object…
Two-Tailed Averaging improves generalization by optimizing the number of leading iterates to ignore.
We propose methods for distributed graph-based multi-task learning that are based on weighted averaging of messages from other machines. Uniform averaging or diminishing stepsize in these methods would yield consensus (single task) learning. We show how simply skewing the averaging weights or controlling the stepsize a…
Stochastic algo learns from evolving data, achieving optimal performance.
Novel approach simplifies VI problems with faster performance.
SGD (Stochastic Gradient Descent) is a popular algorithm for large scale optimization problems due to its low iterative cost. However, SGD can not achieve linear convergence rate as FGD (Full Gradient Descent) because of the inherent gradient variance. To attack the problem, mini-batch SGD was proposed to get a trade-o…
SGDM accelerates faster than SGD with large batch sizes and permits broader learning rates.
We consider in this work a system of two stochastic differential equations named the perturbed compositional gradient flow. By introducing a separation of fast and slow scales of the two equations, we show that the limit of the slow motion is given by an averaged ordinary differential equation. We then demonstrate that…
Stochastic variance reduction algorithms have recently become popular for minimizing the average of a large, but finite number of loss functions. The present paper proposes a Riemannian stochastic quasi-Newton algorithm with variance reduction (R-SQN-VR). The key challenges of averaging, adding, and subtracting multipl…