Kaczmarz++ accelerates convergence for ill-conditioned systems.
problem Solving ill-conditioned linear systems efficiently.
method Adaptive momentum acceleration, Tikhonov-regularized projections, and memoization.
result Kaczmarz++ converges faster than Krylov methods on ill-conditioned systems.
The paper improves Kaczmarz algorithm with momentum for linear least squares.
problem Improving convergence of the Kaczmarz algorithm for linear least squares.
method Integrates geometrically smoothed momentum into the randomized Kaczmarz algorithm.
result Proves expected error reduction in singular vector directions.
The Kaczmarz algorithm is popular for iteratively solving an overdetermined system of linear equations. The traditional Kaczmarz algorithm can approximate the solution in few sweeps through the equations but a randomized version of the Kaczmarz algorithm was shown to converge exponentially and independent of number of …
A new method for LDA using randomized Kaczmarz improves accuracy for large datasets.
problem Efficiently performing LDA on large datasets.
method Randomized Kaczmarz method applied to linear discriminant analysis.
result The method achieves comparable accuracy to full data LDA.
Paper analyzes Hit-and-Run's convergence rates and applies similar methods to randomized Kaczmarz.
problem Quantifying advantages of Hit-and-Run's coordinate-free property.
method Sharp estimates via coupling methods and mixing time bounds.
result Ballistic and superdiffusive convergence rates in certain settings.
New method tackles tensor regression with robust Kaczmarz approach.
problem Reconstructing tensor signals from corrupted measurements.
method Quantile-based randomized Kaczmarz method for tensor linear systems.
result Improved convergence and robustness to adversarial corruptions.
Unified framework for solving linear systems with improved convergence rates.
problem Efficiently solving linear systems with randomized batch-sampling methods.
method Developed a unified randomized batch-sampling Kaczmarz framework with concentration inequalities for analysis.
result Derived new expected linear convergence rate bounds that are tighter and more reflective of empirical behavior.
New study shows faster convergence of SGD and Kaczmarz methods.
problem Improving convergence rates of iterative linear system solvers.
method Last-iterate convergence analysis of SGD with greedy step size over smooth quadratics.
result The t-th iterate attains an O(1/t3/4) convergence rate. We obtain an improved finite-sample guarantee on the linear convergence of stochastic gradient descent for smooth and strongly convex objectives, improving from a quadratic dependence on the conditioning (L/μ)2 (where L is a bound on the smoothness and μ on the strong convexity) to a linear dependence on L/μ. …
This paper is about randomized iterative algorithms for solving a linear system of equations Xβ=y in different settings. Recent interest in the topic was reignited when Strohmer and Vershynin (2009) proved the linear convergence rate of a Randomized Kaczmarz (RK) algorithm that works on the rows of X (data points…
We revisit the problem of inferring the overall ranking among entities in the framework of Bradley-Terry-Luce (BTL) model, based on available empirical data on pairwise preferences. By a simple transformation, we can cast the problem as that of solving a noisy linear system, for which a ready algorithm is available in …
In this paper we present a convergence rate analysis of inexact variants of several randomized iterative methods. Among the methods studied are: stochastic gradient descent, stochastic Newton, stochastic proximal point and stochastic subspace ascent. A common feature of these methods is that in their update rule a cert…
Paper provides linear convergence guarantees for KZIHT and KZPT methods.
problem Solving linear equation systems with sparse constraints.
method Combines Kaczmarz and iterative thresholding methods, using reshuffling data sampling.
result KZIHT and KZPT converge linearly to sparse solutions.
New algorithms improve machine learning performance with explicit regret bounds.
problem Improving machine learning performance with explicit regret bounds.
method Projection-based linear regression algorithms with a focus on modern machine-learning models and their algorithmic performance.
result Established a priori regret bounds with explicit λ-dependence.
New algorithm reduces bias and variance in weighted least-squares solutions.
problem Inconsistent linear least-squares problems with rapidly decaying singular values.
method Regularized block Kaczmarz (ReBlocK) algorithm.
result ReBlocK outperforms RBK and minibatch SGD for inconsistent problems.
AI models solved the Kaczmarz algorithm's worst-case complexity.
problem Finding the worst-case complexity of the Kaczmarz algorithm.
method Combining AI models to analyze the Kaczmarz algorithm's performance.
result Discovered the worst-case complexity of the Kaczmarz algorithm.
We propose a methodology for computing single and multi-asset European option prices, and more generally expectations of scalar functions of (multivariate) random variables. This new approach combines the ability of Monte Carlo simulation to handle high-dimensional problems with the efficiency of function approximation…
Randomized algorithms that base iteration-level decisions on samples from some pool are ubiquitous in machine learning and optimization. Examples include stochastic gradient descent and randomized coordinate descent. This paper makes progress at theoretically evaluating the difference in performance between sampling wi…
New method shows stochastic momentum can converge quickly on optimization problems.
problem Improving convergence of stochastic optimization methods.
method Stochastic heavy ball momentum with minibatching.
result Stochastic heavy ball momentum retains fast linear rate on quadratic problems.
New approach connects stochastic gradient descent to ODE splitting schemes.
problem Improving convergence in stochastic optimization.
method Connection between stochastic gradient descent and ODE splitting schemes.
result Derive a new upper bound on global splitting error.
In recent literature, a general two step procedure has been formulated for solving the problem of phase retrieval. First, a spectral technique is used to obtain a constant-error initial estimate, following which, the estimate is refined to arbitrary precision by first-order optimization of a non-convex loss function. N…
The paper sharpens the analysis of sketch-and-project methods using randomized singular value decomposition.
problem Improving convergence rates of sketch-and-project methods for solving linear systems and non-linear optimization problems.
method Developing a theoretical framework and new spectral bounds for the expected sketched projection matrix.
result The convergence rate improves linearly with sketch size and even faster with certain spectral decays.
Improved rates for continual learning using SGD and last-iterate analysis.
problem Forgetting in overparameterized models after fitting multiple tasks.
method Developed novel SGD upper bounds for continual linear models and analyzed their performance.
result Established universal forgetting rates for continual learning.
We study the phase retrieval problem, which solves quadratic system of equations, i.e., recovers a vector x∈Rn from its magnitude measurements yi=∣⟨ai,x⟩∣,i=1,...,m. We develop a gradient-like algorithm (referred to as RWF representing reshaped W…
SGD achieves near optimal convergence rate in smooth interpolation regime.
problem Optimization of smooth convex objectives with zero noise at optimum.
method Stochastic Gradient Descent (SGD) with large stepsize analysis.
result Last iterate of SGD achieves expected excess risk of O(1/T + σ* / √T) with optimal stepsize.
The study explores how Transformers predict the next token in a sequence.
problem Understanding the mechanism behind Transformers' autoregressive learning ability.
method Exploring the approximation ability of Transformers for next-token prediction through specific instances and a causal kernel descent method.
result Transformer models can learn context-dependent functions f for next-token prediction based on past and current observations. This paper surveys various methods for dimensionality reduction and nearest neighbor search.
problem Efficiently reducing high-dimensional data to lower dimensions while preserving essential information.
method Linear and nonlinear random projections, including sparse random projections, random Fourier Features, and Random Kitchen Sinks.
result Various methods for dimensionality reduction and nearest neighbor search are explained and compared.
The paper studies how norms of random vectors are preserved by random projections.
problem Understanding how random matrix affects norms of random vectors.
method Proved the distribution of the norm of random vector is preserved by random projection.
result Random matrix preserves the distribution of the norm of random vectors with i.i.d. entries.
Orthogonal random features approximate a Bessel kernel, offering sharper bounds than random Fourier features.
problem Approximating Gaussian kernel efficiently for large datasets.
method Use of Haar orthogonal matrices to construct orthogonal random features and analyze their bias and variance.
result Orthogonal random features approximate a Bessel kernel, not the Gaussian kernel, with sharper bounds.
New methods improve prediction performance and reduce computation time in boosting and random forest models.
problem Improving prediction performance and reducing computation time in boosting and random forest models.
method Random tree depth injection approach for Boosting and Random Forests.
result The new methods can improve prediction performance and reduce computation time by up to 40%.
Proposes a method for modeling random objects in metric spaces using random effects.
problem Modeling random objects in non-Euclidean spaces with random effects.
method Nonlinear Fréchet-based algorithm for M-estimation.
result Consistent estimation of prediction target under random-effects formulation.
Enhances random forest performance with exogenous randomness.
problem Improving random forest performance through exogenous randomness.
method Developed non-asymptotic MSE expansions for individual trees and forests, identified two types of randomness, and conducted simulations.
result Exogenous randomness, particularly feature subsampling, reduces both bias and variance of random forests.
New random forest method provides optimal rates and confidence bands.
problem Improving random forest regression rates and constructing confidence bands.
method Proposed Ehrenfest centered purely random forests achieve optimal rates; used Gaussian approximation for supremum of empirical processes.
result Explicit asymptotic uniform confidence bands constructed for both random forest types.
Much is known about random right-angled Coxeter groups (i.e., right-angled Coxeter groups whose defining graphs are random graphs under the Erdös-Rényi model). In this paper, we extend this model to study random general Coxeter groups and give some results about random Coxeter groups, including some information about t…
Improved random forest proximities capture data geometry.
problem Inaccurate random forest proximities do not reflect learned data geometry.
method Introduce RF-GAP: Geometry- and Accuracy-Preserving proximities.
result RF-GAP improves geometric representation in tasks like data imputation.
This work estimates edge weights of edge-reinforced random walks using observed data.
problem Statistical estimation of edge weights in edge-reinforced random walks.
method Proposes an estimator based on the generalized method of moments using the magic formula and hyperbolic Gaussian structure.
result Analyzes the sample complexity of the proposed estimator.
Paper proposes diagnostics for error and variance estimation in randomized matrix computations.
problem Safe use of randomized matrix algorithms in applications.
method Leave-one-out error estimator and jackknife resampling method.
result Provides rapid diagnostics to assess quality of randomized matrix computations.
Random forests reduce bias and variance, especially in low SNR settings.
problem Reducing bias and variance in machine learning models, particularly in low SNR scenarios.
method Empirical study of random forests and bagging ensembles, focusing on the importance of mtry tuning. result Random forests reduce both bias and variance, outperforming bagging ensembles in high SNR settings.
Improved random forest models enhance machine learning predictions.
problem Equal weights for random forest base decision trees are not optimal.
method Proposes algorithms to modify weighting strategy of regular random forest.
result Numerical results show significant improvements over regular random forest.
Study finds no significant difference in neural network weights with quantum random numbers.
problem Effects of biased quantum random numbers on neural network initialization.
method Empirical study using quantum hardware and classical pseudo-random numbers.
result No statistically significant difference found between quantum random numbers and other types.
New method optimizes hyperparameters for randomized algorithms like random feature regression.
problem Optimizing hyperparameters in randomized algorithms is challenging due to their stochastic nature.
method Introduced a random objective function and used ensemble Kalman inversion (EKI) for gradient-free optimization.
result Demonstrated successful optimization of hyperparameters in various randomized algorithms.
In this work we study the degree distribution, the maximum vertex and edge flow in non-uniform random Delaunay triangulations when geodesic routing is used. We also investigate the vertex and edge flow in Erdös-Renyi random graphs, geometric random graphs, expanders and random k-regular graphs. Moreover we show that …
Ensembles of randomized decision trees, usually referred to as random forests, are widely used for classification and regression tasks in machine learning and statistics. Random forests achieve competitive predictive performance and are computationally efficient to train and test, making them excellent candidates for r…
Surveying random sections on Kähler manifolds, leading to metrics.
problem Understanding statistics of random sections on Kähler manifolds.
method Analyzing tensor powers of line bundles.
result Induced metrics from random sections.
The study finds effective lower bounds for spectra of random surfaces and bundles.
problem Determining the spectrum of Laplacian on random surfaces and bundles.
method Analysis of random covering surfaces and unitary bundles over finite-area non-compact hyperbolic surfaces.
result With high probability, the spectrum of random surfaces and bundles has no eigenvalues below a certain threshold.
We review recent advances on the record statistics of strongly correlated time series, whose entries denote the positions of a random walk or a Lévy flight on a line. After a brief survey of the theory of records for independent and identically distributed random variables, we focus on random walks. During the last few…
This study examines how randomness affects machine learning model performance.
problem The impact of randomness on machine learning model performance.
method Empirical study comparing randomness in model training and dataset partitioning.
result Randomness in model training causes more variation in FFNNs than tree-based methods.
We consider a random link, which is defined as the closure of a braid obtained from a random walk on the braid group. For such a random link, the expected value for the number of components was calculated by Jiming Ma. In this paper, we determine the most expected number of components for a random link, and further, co…