The computational and storage complexity of kernel machines presents the primary barrier to their scaling to large, modern, datasets. A common way to tackle the scalability issue is to use the conjugate gradient algorithm, which relieves the constraints on both storage (the kernel matrix need not be stored) and computa…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “preconditioned conjugate gradient”
Two methods solve kernel ridge regression problems efficiently.
problem Solving kernel ridge regression problems with large datasets.
method RPCholesky and KRILL preconditioning techniques.
result Efficient solutions to KRR problems with strong guarantees.
In this paper we introduce a parameter dependent class of Krylov-based methods, namely CD, for the solution of symmetric linear systems. We give evidence that in our proposal we generate sequences of conjugate directions, extending some properties of the standard Conjugate Gradient (CG) method, in order to preserve the…
NeuralIF uses neural networks to improve preconditioning for faster CG convergence.
problem Improving convergence of conjugate gradient method for large-scale sparse systems.
method Data-driven approach using graph neural networks to generate incomplete factorization.
result Data-driven preconditioners accelerate convergence of conjugate gradient method.
We propose a novel Riemannian manifold preconditioning approach for the tensor completion problem with rank constraint. A novel Riemannian metric or inner product is proposed that exploits the least-squares structure of the cost function and takes into account the structured symmetry that exists in Tucker decomposition…
Novel probabilistic solver speeds up solving related linear systems.
problem Efficiently solving multiple related linear systems.
method Probabilistic linear solver over the parameter space, leveraging solved systems.
result Faster and more efficient solution of related linear systems.
Nonlinear conjugate gradient (NLCG) based optimizers have shown superior loss convergence properties compared to gradient descent based optimizers for traditional optimization problems. However, in Deep Neural Network (DNN) training, the dominant optimization algorithm of choice is still Stochastic Gradient Descent (SG…
Deep Learning of Preconditioners for Conjugate Gradient Solvers in Urban Water Related Problemscs.LG
Solving systems of linear equations is a problem occuring frequently in water engineering applications. Usually the size of the problem is too large to be solved via direct factorization. One can resort to iterative approaches, in particular the conjugate gradients method if the matrix is symmetric positive definite. P…
Nowadays stochastic approximation methods are one of the major research direction to deal with the large-scale machine learning problems. From stochastic first order methods, now the focus is shifting to stochastic second order methods due to their faster convergence and availability of computing resources. In this pap…
Sketching techniques have become popular for scaling up machine learning algorithms by reducing the sample size or dimensionality of massive data sets, while still maintaining the statistical power of big data. In this paper, we study sketching from an optimization point of view: we first show that the iterative Hessia…
We present both, theory and an algorithm for solving time-harmonic wave problems in a general setting. The time-harmonic solutions will be achieved by computing time-periodic solutions of the original wave equations. Thus, an exact controllability technique is proposed to solve the time-dependent wave equations. We dis…
Preconditioned non-convex gradient descent improves noisy matrix estimation.
problem Estimating low-rank matrices from noisy measurements.
method Preconditioned non-convex gradient descent for noisy measurements.
result Preconditioned method converges to minimax optimal estimate at a linear rate.
PolarGrad optimizes deep learning models by considering matrix structure, outperforming Adam and Muon.
problem Efficient optimization of large-scale neural networks and language models.
method A unifying framework for analyzing matrix-aware preconditioned methods, including PolarGrad.
result PolarGrad outperforms Adam and Muon in various tasks.
In a modern observational study based on healthcare databases, the number of observations and of predictors typically range in the order of ~ and of ~ . Despite the large sample size, data rarely provide sufficient information to reliably estimate such a large number of parameters. Sparse reg…
A new metric learning framework for signed graphs using Gershgorin disc alignment.
problem Learning Mahalanobis metrics from signed graphs efficiently.
method Proposes a fast metric learning framework using Gershgorin disc perfect alignment (GDPA) to circumvent full eigen-decomposition.
result Proves that Gershgorin disc left-ends of similarity transform are perfectly aligned at the smallest eigenvalue, enabling efficient optimization.
The task of choosing a preconditioner to use when solving a linear system with iterative methods is difficult. For instance, even if one has access to a collection of candidate preconditioners, it is currently …
Optimal preconditioning improves Langevin sampling efficiency.
problem Improving sampling efficiency in high-dimensional target distributions.
method Optimal preconditioning using Fisher information, applied to MALA.
result Adaptive MCMC scheme significantly outperforms other methods.
Preconditioned gradient methods are among the most general and powerful tools in optimization. However, preconditioning requires storing and manipulating prohibitively large matrices. We describe and analyze a new structure-aware preconditioning algorithm, called Shampoo, for stochastic optimization over tensor spaces.…
Bias correction improves language model training performance.
problem Stochastic update bias in preconditioned optimizers.
method Cross-fitted preconditioning and variance-corrected inversion.
result Reduces held-out pretraining loss by 0.15 nats.
Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) is demonstrated to efficiently solve eigenvalue problems for graph Laplacians that appear in spectral clustering. For static graph partitioning, 10-20 iterations of LOBPCG without preconditioning result in ~10x error reduction, enough to achieve 100% corr…
Stochastic gradient descent improves Gaussian process regression.
problem Efficiently solving large linear systems in Gaussian process regression.
method Developed a stochastic dual descent algorithm using insights from optimisation and kernel communities.
result Stochastic gradient descent is highly effective when done right.
Gradient-based MCMC for discrete spaces improves sampling performance.
problem Sampling in discrete spaces using traditional methods is challenging.
method Introduced new discrete Metropolis-Hastings samplers inspired by MALA, with a novel preconditioning technique.
result Demonstrated strong empirical performance across various challenging sampling problems.
Improved convergence for overparameterized low-rank matrix sensing.
problem Overparameterized low-rank matrix sensing with unknown rank and ill-conditioning.
method ScaledGD() - preconditioned gradient descent method.
result ScaledGD() converges at a constant linear rate after a logarithmic number of iterations.
We provide an online convex optimization algorithm with regret that interpolates between the regret of an algorithm using an optimal preconditioning matrix and one using a diagonal preconditioning matrix. Our regret bound is never worse than that obtained by diagonal preconditioning, and in certain setting even surpass…
Stochastic Gradient Langevin Dynamics infuses isotropic gradient noise to SGD to help navigate pathological curvature in the loss landscape for deep networks. Isotropic nature of the noise leads to poor scaling, and adaptive methods based on higher order curvature information such as Fisher Scoring have been proposed t…
Dual Space Preconditioning speeds up gradient descent in overparameterized models.
problem Improving convergence of gradient descent in overparameterized linear models.
method Introducing a novel preconditioner of the form for convex and applying it to overparameterized linear models.
result The iterates of the preconditioned gradient descent converge to a solution satisfying .
Adaptive learning rate algorithms such as RMSProp are widely used for training deep neural networks. RMSProp offers efficient training since it uses first order gradients to approximate Hessian-based preconditioning. However, since the first order gradients include noise caused by stochastic optimization, the approxima…
Stochastic gradient descent (SGD) still is the workhorse for many practical problems. However, it converges slow, and can be difficult to tune. It is possible to precondition SGD to accelerate its convergence remarkably. But many attempts in this direction either aim at solving specialized problems, or result in signif…
A new ML method speeds up PDE simulations without needing classical training.
problem Accelerating transient PDE simulations using machine learning.
method Online-learned preconditioners using a bandit algorithm.
result One-shot acceleration of PDE simulations.
Stochastic Gradient Descent in Hilbert Scales: Smoothness, Preconditioning and Earlier Stoppingstat.ML
Stochastic Gradient Descent improved for various Hilbert scales and misspecified models.
problem Understanding and optimizing SGD in Hilbert scales for machine learning.
method Extending SGD analysis to Hilbert scales, including Sobolev and Diffusion spaces, and showing the effects of smoothness and preconditioning.
result Violation of smoothness assumption affects learning rate; preconditioning in Hilbert scales reduces the number of iterations for misspecified models.
Bayesian sparse learning method improves deep neural network efficiency.
problem Sparse learning in deep neural networks with complex geometry.
method Preconditioned stochastic gradient Langevin Dynamics (PSGLD) for sampling and adaptive optimization of hyperparameters.
result The proposed algorithm achieves asymptotic convergence with controlled bias.
TDprop uses Jacobi preconditioning to improve adaptive optimizers in Deep RL.
problem Improving performance of adaptive optimizers in Deep RL.
method TDprop computes per-parameter learning rates based on Jacobi preconditioning of the TD update rule.
result TDprop matches or exceeds Adam's performance in Deep RL experiments, suggesting Jacobi preconditioning can improve adaptive methods.
Sequential Least-Squares Estimators with Fast Randomized Sketching for Linear Statistical Modelsstat.ML
A new method for estimating large-scale linear models with improved precision.
problem Estimating large-scale linear statistical models efficiently.
method Sequential Least-Squares Estimators with Fast Randomized Sketching (SLSE-FRS), integrating Sketch-and-Solve and Iterative-Sketching methods.
result SLSE-FRS produces high-precision estimators, outperforming state-of-the-art methods.
Standard gradient descent methods are susceptible to a range of issues that can impede training, such as high correlations and different scaling in parameter space.These difficulties can be addressed by second-order approaches that apply a pre-conditioning matrix to the gradient to improve convergence. Unfortunately, s…
This paper studies the performance of a recently proposed preconditioned stochastic gradient descent (PSGD) algorithm on recurrent neural network (RNN) training. PSGD adaptively estimates a preconditioner to accelerate gradient descent, and is designed to be simple, general and easy to use, as stochastic gradient desce…
Randomized block-diagonal preconditioning improves parallel learning convergence.
problem Improving convergence of gradient-based optimization methods in parallel settings.
method Randomization of coordinates during optimization to repartition tasks.
result Randomization significantly improves convergence of block-diagonal preconditioned methods.
Effective training of deep neural networks suffers from two main issues. The first is that the parameter spaces of these models exhibit pathological curvature. Recent methods address this problem by using adaptive preconditioning for Stochastic Gradient Descent (SGD). These methods improve convergence by adapting to th…
Researchers compare different gradient methods for ridge regression, finding conjugate gradients have similar performance.
problem Comparing statistical properties of different gradient methods in ridge regression.
method Explicit non-standard error decomposition to bound prediction error of conjugate gradient iterates.
result Conjugate gradient iterates share optimality properties with gradient flow and ridge regression up to a constant factor.
Gradient descent with preconditioning finds global optima in overparameterized nonconvex factorization.
problem Finding global optima in nonconvex Burer-Monteiro factorization.
method Preconditioned gradient descent for overparameterized nonconvex function minimization.
result Gradient descent with preconditioning achieves linear convergence in the overparameterized case.
Unified framework for understanding and optimizing training acceleration.
problem Challenges in optimizing training with regularization and acceleration techniques.
method Explains how AdaGrad, RMSProp, and Adam accelerate training, and derives a generalization for -regularization.
result Derives a unified mathematical framework for understanding and optimizing training acceleration.
APGD algorithm efficiently recovers over-parameterized matrices from noisy measurements.
problem Matrix sensing problem with over-parameterization and noisy measurements.
method Alternating preconditioned gradient descent (APGD) algorithm incorporating preconditioning terms.
result APGD converges to a near-optimal error at a linear rate.
New stability analysis improves generalization of multipass SGD.
problem Improper preconditioning affects generalization in multipass SGD.
method Developed on-average stability analysis for multipass SGD.
result Proper preconditioning yields optimal effective dimension dependence.
Scalable Krylov Subspace Methods for Generalized Mixed-Effects Models with Crossed Random Effectsstat.ME
New Krylov subspace methods speed up mixed-effects models with crossed random effects.
problem Slow computations for high-dimensional crossed random effects in mixed-effects models.
method Krylov subspace-based methods for generalized mixed-effects models with cross effects.
result Speedups by factors of up to 10,000 in computations for mixed-effects models.
Stein variational gradient descent (SVGD) is a particle-based inference algorithm that leverages gradient information for efficient approximate inference. In this work, we enhance SVGD by leveraging preconditioning matrices, such as the Hessian and Fisher information matrix, to incorporate geometric information into SV…
Polyak-Ruppert CLT for SA-Adam with momentum and non-convergent adaptive preconditioning
problem Adaptive optimizers combining momentum and non-convergent preconditioning
method Proving positive drift stability and a non-autonomous Polyak-Ruppert CLT for SA-Adam
result The iterate-marginal covariance is exactly the plain stochastic gradient descent (SGD) sandwich
Increasing the batch size is a popular way to speed up neural network training, but beyond some critical batch size, larger batch sizes yield diminishing returns. In this work, we study how the critical batch size changes based on properties of the optimization algorithm, including acceleration and preconditioning, thr…
Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Modelsstat.ML
The natural gradient method has been used effectively in conjugate Gaussian process models, but the non-conjugate case has been largely unexplored. We examine how natural gradients can be used in non-conjugate stochastic settings, together with hyperparameter learning. We conclude that the natural gradient can signific…
SignSGD analysis quantifies its effects in high dimensions.
problem Understanding signSGD's effects in high-dimensional settings.
method High-dimensional analysis of signSGD, deriving SDE and ODE for risk.
result Quantification of signSGD's effects: effective learning rate, noise compression, diagonal preconditioning, gradient noise reshaping.