The paper improves Kaczmarz algorithm with momentum for linear least squares.
problem Improving convergence of the Kaczmarz algorithm for linear least squares.
method Integrates geometrically smoothed momentum into the randomized Kaczmarz algorithm.
result Proves expected error reduction in singular vector directions.
The Kaczmarz algorithm is popular for iteratively solving an overdetermined system of linear equations. The traditional Kaczmarz algorithm can approximate the solution in few sweeps through the equations but a randomized version of the Kaczmarz algorithm was shown to converge exponentially and independent of number of …
Kaczmarz++ accelerates convergence for ill-conditioned systems.
problem Solving ill-conditioned linear systems efficiently.
method Adaptive momentum acceleration, Tikhonov-regularized projections, and memoization.
result Kaczmarz++ converges faster than Krylov methods on ill-conditioned systems.
A new method for LDA using randomized Kaczmarz improves accuracy for large datasets.
problem Efficiently performing LDA on large datasets.
method Randomized Kaczmarz method applied to linear discriminant analysis.
result The method achieves comparable accuracy to full data LDA.
Paper analyzes Hit-and-Run's convergence rates and applies similar methods to randomized Kaczmarz.
problem Quantifying advantages of Hit-and-Run's coordinate-free property.
method Sharp estimates via coupling methods and mixing time bounds.
result Ballistic and superdiffusive convergence rates in certain settings.
New method tackles tensor regression with robust Kaczmarz approach.
problem Reconstructing tensor signals from corrupted measurements.
method Quantile-based randomized Kaczmarz method for tensor linear systems.
result Improved convergence and robustness to adversarial corruptions.
This paper is about randomized iterative algorithms for solving a linear system of equations Xβ=y in different settings. Recent interest in the topic was reignited when Strohmer and Vershynin (2009) proved the linear convergence rate of a Randomized Kaczmarz (RK) algorithm that works on the rows of X (data points…
New study shows faster convergence of SGD and Kaczmarz methods.
problem Improving convergence rates of iterative linear system solvers.
method Last-iterate convergence analysis of SGD with greedy step size over smooth quadratics.
result The t-th iterate attains an O(1/t3/4) convergence rate. We obtain an improved finite-sample guarantee on the linear convergence of stochastic gradient descent for smooth and strongly convex objectives, improving from a quadratic dependence on the conditioning (L/μ)2 (where L is a bound on the smoothness and μ on the strong convexity) to a linear dependence on L/μ. …
We revisit the problem of inferring the overall ranking among entities in the framework of Bradley-Terry-Luce (BTL) model, based on available empirical data on pairwise preferences. By a simple transformation, we can cast the problem as that of solving a noisy linear system, for which a ready algorithm is available in …
Unified framework for solving linear systems with improved convergence rates.
problem Efficiently solving linear systems with randomized batch-sampling methods.
method Developed a unified randomized batch-sampling Kaczmarz framework with concentration inequalities for analysis.
result Derived new expected linear convergence rate bounds that are tighter and more reflective of empirical behavior.
New algorithms improve machine learning performance with explicit regret bounds.
problem Improving machine learning performance with explicit regret bounds.
method Projection-based linear regression algorithms with a focus on modern machine-learning models and their algorithmic performance.
result Established a priori regret bounds with explicit λ-dependence.
New algorithm reduces bias and variance in weighted least-squares solutions.
problem Inconsistent linear least-squares problems with rapidly decaying singular values.
method Regularized block Kaczmarz (ReBlocK) algorithm.
result ReBlocK outperforms RBK and minibatch SGD for inconsistent problems.
AI models solved the Kaczmarz algorithm's worst-case complexity.
problem Finding the worst-case complexity of the Kaczmarz algorithm.
method Combining AI models to analyze the Kaczmarz algorithm's performance.
result Discovered the worst-case complexity of the Kaczmarz algorithm.
New method for option pricing using Monte Carlo and least squares.
problem Computing European option prices in high dimensions.
method Combines Monte Carlo simulation with least squares approximation and randomized Kaczmarz algorithm.
result Efficient method for high-dimensional integration and option pricing.
Online SGD from random init solves non-smooth, non-convex phase retrieval.
problem Solving phase retrieval with non-smooth, non-convex loss functions.
method Online stochastic gradient descent (SGD) with constant step size, starting from arbitrary initialization.
result SGD converges from arbitrary initializations for the amplitude squared loss objective.
In this paper we present a convergence rate analysis of inexact variants of several randomized iterative methods. Among the methods studied are: stochastic gradient descent, stochastic Newton, stochastic proximal point and stochastic subspace ascent. A common feature of these methods is that in their update rule a cert…
Paper provides linear convergence guarantees for KZIHT and KZPT methods.
problem Solving linear equation systems with sparse constraints.
method Combines Kaczmarz and iterative thresholding methods, using reshuffling data sampling.
result KZIHT and KZPT converge linearly to sparse solutions.
Randomized algorithms that base iteration-level decisions on samples from some pool are ubiquitous in machine learning and optimization. Examples include stochastic gradient descent and randomized coordinate descent. This paper makes progress at theoretically evaluating the difference in performance between sampling wi…
New method shows stochastic momentum can converge quickly on optimization problems.
problem Improving convergence of stochastic optimization methods.
method Stochastic heavy ball momentum with minibatching.
result Stochastic heavy ball momentum retains fast linear rate on quadratic problems.
New approach connects stochastic gradient descent to ODE splitting schemes.
problem Improving convergence in stochastic optimization.
method Connection between stochastic gradient descent and ODE splitting schemes.
result Derive a new upper bound on global splitting error.
The paper sharpens the analysis of sketch-and-project methods using randomized singular value decomposition.
problem Improving convergence rates of sketch-and-project methods for solving linear systems and non-linear optimization problems.
method Developing a theoretical framework and new spectral bounds for the expected sketched projection matrix.
result The convergence rate improves linearly with sketch size and even faster with certain spectral decays.
We study the phase retrieval problem, which solves quadratic system of equations, i.e., recovers a vector x∈Rn from its magnitude measurements yi=∣⟨ai,x⟩∣,i=1,...,m. We develop a gradient-like algorithm (referred to as RWF representing reshaped W…
Improved rates for continual learning using SGD and last-iterate analysis.
problem Forgetting in overparameterized models after fitting multiple tasks.
method Developed novel SGD upper bounds for continual linear models and analyzed their performance.
result Established universal forgetting rates for continual learning.
SGD achieves near optimal convergence rate in smooth interpolation regime.
problem Optimization of smooth convex objectives with zero noise at optimum.
method Stochastic Gradient Descent (SGD) with large stepsize analysis.
result Last iterate of SGD achieves expected excess risk of O(1/T + σ* / √T) with optimal stepsize.
The study explores how Transformers predict the next token in a sequence.
problem Understanding the mechanism behind Transformers' autoregressive learning ability.
method Exploring the approximation ability of Transformers for next-token prediction through specific instances and a causal kernel descent method.
result Transformer models can learn context-dependent functions f for next-token prediction based on past and current observations. Study on randomized algorithms for optimal stopping problems.
problem Optimal stopping problems in randomized algorithms.
method Forward and backward Monte Carlo based optimisation algorithms.
result Proved convergence of the proposed algorithms and derived convergence rates.
Paper proposes diagnostics for error and variance estimation in randomized matrix computations.
problem Safe use of randomized matrix algorithms in applications.
method Leave-one-out error estimator and jackknife resampling method.
result Provides rapid diagnostics to assess quality of randomized matrix computations.
New method optimizes hyperparameters for randomized algorithms like random feature regression.
problem Optimizing hyperparameters in randomized algorithms is challenging due to their stochastic nature.
method Introduced a random objective function and used ensemble Kalman inversion (EKI) for gradient-free optimization.
result Demonstrated successful optimization of hyperparameters in various randomized algorithms.
We propose accelerated randomized coordinate descent algorithms for stochastic optimization and online learning. Our algorithms have significantly less per-iteration complexity than the known accelerated gradient algorithms. The proposed algorithms for online learning have better regret performance than the known rando…
New random forest algorithm improves regression with missing data.
problem Regression with missing data values.
method New random forest algorithm compared to existing techniques.
result Improved performance in quadratic errors and bias compared to existing methods.
This paper compares Grid Search, Random Search, and Genetic Algorithm for NAS.
problem Hyperparameter optimization for neural architecture search.
method Comparison of Grid Search, Random Search, and Genetic Algorithm.
result Genetic Algorithm outperforms Grid Search and Random Search in terms of accuracy and execution time.
Paper studies randomized spectral clustering for large-scale networks.
problem Computational challenges in large-scale network community detection.
method Randomized sketching algorithms for spectral clustering.
result Theoretical bounds for approximation, misclassification, and link probability estimation.
Improved matrix approximation using randomized algorithms.
problem Finding better approximations of given matrices.
method Randomized algorithms to compute (HT) as an improved approximation. result Computed (HT) provides a better approximation than given F∗. This study examines how randomness affects machine learning model performance.
problem The impact of randomness on machine learning model performance.
method Empirical study comparing randomness in model training and dataset partitioning.
result Randomness in model training causes more variation in FFNNs than tree-based methods.
HD algorithm simulates dynamics on random matrix ensembles without generating full matrices.
problem Simulating dynamics on dense random matrix ensembles with high space and time complexity.
method Householder reflectors for adaptive and recursive construction, deferring decisions.
result Significant reductions in runtime and memory footprint for practical T≪n. New MBQC algorithm uses randomness for generative modeling.
problem Designing efficient quantum algorithms for generative modeling.
method Proposes a variational MBQC algorithm that treats randomness as a resource.
result Randomness in MBQC can lead to significant gains in generative modeling performance.
Polynomial-time algorithm estimates edge density of random graphs with privacy and robustness.
problem Estimating edge density of random graphs while maintaining privacy and robustness.
method Sum-of-squares algorithm for robust edge density estimation and reduction from privacy to robustness.
result Optimal error rate up to logarithmic factors, matching theoretical lower bounds.
Gradient span algorithms show consistent progress in high dimensions.
problem Understanding consistent training progress in large machine learning models.
method Proving deterministic behavior of gradient span algorithms on Gaussian random functions.
result Gradient span algorithms have asymptotically deterministic behavior in high dimensions.
Despite widespread interest and practical use, the theoretical properties of random forests are still not well understood. In this paper we contribute to this understanding in two ways. We present a new theoretically tractable variant of random regression forests and prove that our algorithm is consistent. We also prov…
New algorithm learns random neural networks efficiently.
problem Learning random constant-depth neural networks efficiently.
method Presented a PTAS (Polynomial-Time Approximation Scheme) for learning random Xavier networks of fixed depth.
result For any fixed ε and depth i, there is a poly-time algorithm that learns random Xavier networks up to an additive error of ε.
Although the methods of bagging and random forests are some of the most widely used prediction methods, relatively little is known about their algorithmic convergence. In particular, there are not many theoretical guarantees for deciding when an ensemble is "large enough" --- so that its accuracy is close to that of an…
Paper reinterprets ARP algorithm and improves its analysis and speed.
problem Column subset selection in data analysis.
method Volume sampling and active learning connections, new analysis, rejection sampling.
result Faster implementations and new analysis for ARP algorithm.
We propose a novel probabilistic method for detection of objects in noisy images. The method uses results from percolation and random graph theories. We present an algorithm that allows to detect objects of unknown shapes in the presence of random noise. The algorithm has linear complexity and exponential accuracy and …
Randomized SINDy learns dynamic data structures using probabilistic methods.
problem Learning time-dependent data structures in dynamic systems.
method Sequential machine learning with a probabilistic approach, incorporating feature augmentation and Tikhonov regularization.
result Demonstrated effectiveness in regression and binary classification using real-world data.
Randomized algorithm solves vector-valued regression problems with low-rank operators.
problem Vector-valued regression problems involving infinite-dimensional spaces.
method Randomized Reduced Rank Regression (R4) using Gaussian sketching for optimization.
result R4 estimators are efficient and accurate, with empirical risk close to optimal.
We consider statistical and algorithmic aspects of solving large-scale least-squares (LS) problems using randomized sketching algorithms. Prior results show that, from an \emph{algorithmic perspective}, when using sketching matrices constructed from random projections and leverage-score sampling, if the number of sampl…
Two ANOVA-based algorithms boost random Fourier feature models for function approximation.
problem Approximating high-dimensional functions with low-order interactions.
method Utilizes ANOVA decomposition to learn low-order functions and index sets of important variables.
result Significantly reduces approximation error compared to existing methods.