arXiv research

Kaczmarz++ accelerates convergence for ill-conditioned systems.

New algorithms improve machine learning performance with explicit regret bounds.

The paper improves Kaczmarz algorithm with momentum for linear least squares.

AI models solved the Kaczmarz algorithm's worst-case complexity.

The Kaczmarz algorithm is popular for iteratively solving an overdetermined system of linear equations. The traditional Kaczmarz algorithm can approximate the solution in few sweeps through the equations but a randomized version of the Kaczmarz algorithm was shown to converge exponentially and independent of number of …

New method tackles tensor regression with robust Kaczmarz approach.

New study shows faster convergence of SGD and Kaczmarz methods.

A new method for LDA using randomized Kaczmarz improves accuracy for large datasets.

Paper analyzes Hit-and-Run's convergence rates and applies similar methods to randomized Kaczmarz.

Unified framework for solving linear systems with improved convergence rates.

Paper provides linear convergence guarantees for KZIHT and KZPT methods.

We obtain an improved finite-sample guarantee on the linear convergence of stochastic gradient descent for smooth and strongly convex objectives, improving from a quadratic dependence on the conditioning $(L/μ)^2$ (where $L$ is a bound on the smoothness and $μ$ on the strong convexity) to a linear dependence on $L/μ$. …

This paper is about randomized iterative algorithms for solving a linear system of equations $X β= y$ in different settings. Recent interest in the topic was reignited when Strohmer and Vershynin (2009) proved the linear convergence rate of a Randomized Kaczmarz (RK) algorithm that works on the rows of $X$ (data points…

We revisit the problem of inferring the overall ranking among entities in the framework of Bradley-Terry-Luce (BTL) model, based on available empirical data on pairwise preferences. By a simple transformation, we can cast the problem as that of solving a noisy linear system, for which a ready algorithm is available in …

New algorithm reduces bias and variance in weighted least-squares solutions.

New approach connects stochastic gradient descent to ODE splitting schemes.

In this paper we present a convergence rate analysis of inexact variants of several randomized iterative methods. Among the methods studied are: stochastic gradient descent, stochastic Newton, stochastic proximal point and stochastic subspace ascent. A common feature of these methods is that in their update rule a cert…

We propose a methodology for computing single and multi-asset European option prices, and more generally expectations of scalar functions of (multivariate) random variables. This new approach combines the ability of Monte Carlo simulation to handle high-dimensional problems with the efficiency of function approximation…

New method shows stochastic momentum can converge quickly on optimization problems.

Randomized algorithms that base iteration-level decisions on samples from some pool are ubiquitous in machine learning and optimization. Examples include stochastic gradient descent and randomized coordinate descent. This paper makes progress at theoretically evaluating the difference in performance between sampling wi…

In recent literature, a general two step procedure has been formulated for solving the problem of phase retrieval. First, a spectral technique is used to obtain a constant-error initial estimate, following which, the estimate is refined to arbitrary precision by first-order optimization of a non-convex loss function. N…

We study the phase retrieval problem, which solves quadratic system of equations, i.e., recovers a vector $\boldsymbol{x}\in \mathbb{R}^n$ from its magnitude measurements $y_i=|\langle \boldsymbol{a}_i, \boldsymbol{x}\rangle|, i=1,..., m$. We develop a gradient-like algorithm (referred to as RWF representing reshaped W…

The study explores how Transformers predict the next token in a sequence.

SGD achieves near optimal convergence rate in smooth interpolation regime.

The paper sharpens the analysis of sketch-and-project methods using randomized singular value decomposition.

Improved rates for continual learning using SGD and last-iterate analysis.