arXiv research

PEARL uses reinforcement learning to improve matrix preconditioners.

New analysis for learning and applying preconditioners in MCMC improves efficiency.

We study two types of preconditioners and preconditioned stochastic gradient descent (SGD) methods in a unified framework. We call the first one the Newton type due to its close relationship to the Newton method, and the second one the Fisher type as its preconditioner is closely related to the inverse of Fisher inform…

Analyzes learning and applying preconditioners in MCMC for efficiency.

New preconditioners speed up SGD on Lie groups.

Shampoo optimizes preconditioners for faster convergence in machine learning.

Nystrom approximation speeds up kernel model training.

We present a novel preconditioning technique for proximal optimization methods that relies on graph algorithms to construct effective preconditioners. Such combinatorial preconditioners arise from partitioning the graph into forests. We prove that certain decompositions lead to a theoretically optimal condition number.…

This paper investigates Shampoo's heuristics and decouples preconditioner updates.

Scalable algorithm for sampling Gaussian processes using sparse grids and preconditioners.

Solving systems of linear equations is a problem occuring frequently in water engineering applications. Usually the size of the problem is too large to be solved via direct factorization. One can resort to iterative approaches, in particular the conjugate gradients method if the matrix is symmetric positive definite. P…

New MCMC method learns sparse preconditioner for high-dimensional problems.

The task of choosing a preconditioner $\boldsymbol{M}$ to use when solving a linear system $\boldsymbol{Ax}=\boldsymbol{b}$ with iterative methods is difficult. For instance, even if one has access to a collection $\boldsymbol{M}_1,\boldsymbol{M}_2,\ldots,\boldsymbol{M}_n$ of candidate preconditioners, it is currently …

This paper proposes a family of online second order methods for possibly non-convex stochastic optimizations based on the theory of preconditioned stochastic gradient descent (PSGD), which can be regarded as an enhance stochastic Newton method with the ability to handle gradient noise and non-convexity simultaneously. …

Adaptive methods such as Adam and RMSProp are widely used in deep learning but are not well understood. In this paper, we seek a crisp, clean and precise characterization of their behavior in nonconvex settings. To this end, we first provide a novel view of adaptive methods as preconditioned SGD, where the precondition…

Stochastic gradient descent (SGD) still is the workhorse for many practical problems. However, it converges slow, and can be difficult to tune. It is possible to precondition SGD to accelerate its convergence remarkably. But many attempts in this direction either aim at solving specialized problems, or result in signif…

RL accelerates portfolio optimization and option pricing by dynamically adjusting preconditioner sizes.

New iterative methods improve scalability of Gaussian process approximations for large data.

NeuralIF uses neural networks to improve preconditioning for faster CG convergence.

New method speeds up kernel-based machine learning for force field reconstruction.

AI-driven framework optimizes MCMC-based preconditioners for faster linear system solving.

Dual Space Preconditioning speeds up gradient descent in overparameterized models.

Muon outperforms GD in associative memory learning by balancing frequency components.

Bias correction improves language model training performance.

New algorithm learns LQR with $O(\sqrt{T})$ regret using Langevin dynamics and excitation.

ASTRA improves TDA by more accurately approximating iHVP.

Unified framework for training neural networks with non-smooth, non-convex regularizers.

This paper optimizes diagonal preconditioning to improve matrix condition numbers.

A new ML method speeds up PDE simulations without needing classical training.

Optimizes nonconvex optimization by converting it to static regret minimization.

New methods improve solving linear systems and preconditioning with reduced complexity.

A new method reduces complexity of normalizing flows for MCMC preconditioning.

The successive projection algorithm (SPA) can quickly solve a nonnegative matrix factorization problem under a separability assumption. Even if noise is added to the problem, SPA is robust as long as the perturbations caused by the noise are small. In particular, robustness against noise should be high when handling th…

New lower bounds show sparse recovery is hard even with multiple preconditioners.

When the design matrix has orthonormal columns, "soft thresholding" the ordinary least squares (OLS) solution produces the Lasso solution [Tibshirani, 1996]. If one uses the Puffer preconditioned Lasso [Jia and Rohe, 2012], then this result generalizes from orthonormal designs to full rank designs (Theorem 1). Theorem …

A new method improves convergence in low-rank approximation.

The study explores various localized bases and their duals for scattered data approximation.

The paper studies efficient Hessian fitting methods for stochastic optimization.

HIP-GP improves GP inference for inter-domain observations with millions of inducing points.

This paper studies the performance of a recently proposed preconditioned stochastic gradient descent (PSGD) algorithm on recurrent neural network (RNN) training. PSGD adaptively estimates a preconditioner to accelerate gradient descent, and is designed to be simple, general and easy to use, as stochastic gradient desce…

LoRA-One uses one-step full gradient to align adapters for efficient large model fine-tuning.

A method to fix radius distortion in generative models on curved spaces.

Optimizer memory affects learning rate sensitivity in shuffle order, impacting fine-tuning noise.

Novel probabilistic solver speeds up solving related linear systems.

SLMC improves sampling efficiency for high-dimensional distributions.

The paper analyzes Adam and SGD in nonstationary optimization, revealing tradeoffs between noise and drift.

New recommendations improve Gaussian process accuracy and stability.

Paper analyzes Langevin dynamics for solving infinite-dimensional Bayesian inverse problems.