arXiv research

PEARL uses reinforcement learning to improve matrix preconditioners.

We study two types of preconditioners and preconditioned stochastic gradient descent (SGD) methods in a unified framework. We call the first one the Newton type due to its close relationship to the Newton method, and the second one the Fisher type as its preconditioner is closely related to the inverse of Fisher inform…

New preconditioners speed up SGD on Lie groups.

The task of choosing a preconditioner $\boldsymbol{M}$ to use when solving a linear system $\boldsymbol{Ax}=\boldsymbol{b}$ with iterative methods is difficult. For instance, even if one has access to a collection $\boldsymbol{M}_1,\boldsymbol{M}_2,\ldots,\boldsymbol{M}_n$ of candidate preconditioners, it is currently …

New analysis for learning and applying preconditioners in MCMC improves efficiency.

Analyzes learning and applying preconditioners in MCMC for efficiency.

Nystrom approximation speeds up kernel model training.

We present a novel preconditioning technique for proximal optimization methods that relies on graph algorithms to construct effective preconditioners. Such combinatorial preconditioners arise from partitioning the graph into forests. We prove that certain decompositions lead to a theoretically optimal condition number.…

Shampoo optimizes preconditioners for faster convergence in machine learning.

Stochastic gradient descent (SGD) still is the workhorse for many practical problems. However, it converges slow, and can be difficult to tune. It is possible to precondition SGD to accelerate its convergence remarkably. But many attempts in this direction either aim at solving specialized problems, or result in signif…

This paper investigates Shampoo's heuristics and decouples preconditioner updates.

Scalable algorithm for sampling Gaussian processes using sparse grids and preconditioners.

This paper proposes a family of online second order methods for possibly non-convex stochastic optimizations based on the theory of preconditioned stochastic gradient descent (PSGD), which can be regarded as an enhance stochastic Newton method with the ability to handle gradient noise and non-convexity simultaneously. …

AI-driven framework optimizes MCMC-based preconditioners for faster linear system solving.

New MCMC method learns sparse preconditioner for high-dimensional problems.

Solving systems of linear equations is a problem occuring frequently in water engineering applications. Usually the size of the problem is too large to be solved via direct factorization. One can resort to iterative approaches, in particular the conjugate gradients method if the matrix is symmetric positive definite. P…

Bias correction improves language model training performance.

Adaptive methods such as Adam and RMSProp are widely used in deep learning but are not well understood. In this paper, we seek a crisp, clean and precise characterization of their behavior in nonconvex settings. To this end, we first provide a novel view of adaptive methods as preconditioned SGD, where the precondition…

New iterative methods improve scalability of Gaussian process approximations for large data.

NeuralIF uses neural networks to improve preconditioning for faster CG convergence.

Dual Space Preconditioning speeds up gradient descent in overparameterized models.

This paper optimizes diagonal preconditioning to improve matrix condition numbers.

RL accelerates portfolio optimization and option pricing by dynamically adjusting preconditioner sizes.

New method speeds up kernel-based machine learning for force field reconstruction.

Unified framework for training neural networks with non-smooth, non-convex regularizers.

Muon outperforms GD in associative memory learning by balancing frequency components.

Optimizes nonconvex optimization by converting it to static regret minimization.

ASTRA improves TDA by more accurately approximating iHVP.

New methods improve solving linear systems and preconditioning with reduced complexity.

New algorithm learns LQR with $O(\sqrt{T})$ regret using Langevin dynamics and excitation.

A new method reduces complexity of normalizing flows for MCMC preconditioning.

The successive projection algorithm (SPA) can quickly solve a nonnegative matrix factorization problem under a separability assumption. Even if noise is added to the problem, SPA is robust as long as the perturbations caused by the noise are small. In particular, robustness against noise should be high when handling th…

New lower bounds show sparse recovery is hard even with multiple preconditioners.

When the design matrix has orthonormal columns, "soft thresholding" the ordinary least squares (OLS) solution produces the Lasso solution [Tibshirani, 1996]. If one uses the Puffer preconditioned Lasso [Jia and Rohe, 2012], then this result generalizes from orthonormal designs to full rank designs (Theorem 1). Theorem …

A new method improves convergence in low-rank approximation.

The study explores various localized bases and their duals for scattered data approximation.

The paper studies efficient Hessian fitting methods for stochastic optimization.

HIP-GP improves GP inference for inter-domain observations with millions of inducing points.

A new ML method speeds up PDE simulations without needing classical training.

LoRA-One uses one-step full gradient to align adapters for efficient large model fine-tuning.

The paper analyzes Adam and SGD in nonstationary optimization, revealing tradeoffs between noise and drift.

Paper analyzes Langevin dynamics for solving infinite-dimensional Bayesian inverse problems.

New iterative methods improve Vecchia-Laplace approximations for large data sets.

A method to fix radius distortion in generative models on curved spaces.

This paper studies the performance of a recently proposed preconditioned stochastic gradient descent (PSGD) algorithm on recurrent neural network (RNN) training. PSGD adaptively estimates a preconditioner to accelerate gradient descent, and is designed to be simple, general and easy to use, as stochastic gradient desce…

Paper analyzes Langevin dynamics for multimodal Gaussian mixtures, controlling errors across dimensions.

New algorithm estimates Gaussian means and covariances efficiently and privately.

State-of-the-art models are now trained with billions of parameters, reaching hardware limits in terms of memory consumption. This has created a recent demand for memory-efficient optimizers. To this end, we investigate the limits and performance tradeoffs of memory-efficient adaptively preconditioned gradient methods.…