arXiv research

PEARL uses reinforcement learning to improve matrix preconditioners.

We study two types of preconditioners and preconditioned stochastic gradient descent (SGD) methods in a unified framework. We call the first one the Newton type due to its close relationship to the Newton method, and the second one the Fisher type as its preconditioner is closely related to the inverse of Fisher inform…

New preconditioners speed up SGD on Lie groups.

NeuralIF uses neural networks to improve preconditioning for faster CG convergence.

AI-driven framework optimizes MCMC-based preconditioners for faster linear system solving.

This paper optimizes diagonal preconditioning to improve matrix condition numbers.

New methods improve solving linear systems and preconditioning with reduced complexity.

Muon outperforms GD in associative memory learning by balancing frequency components.

Shampoo optimizes preconditioners for faster convergence in machine learning.

This paper investigates Shampoo's heuristics and decouples preconditioner updates.

Solving systems of linear equations is a problem occuring frequently in water engineering applications. Usually the size of the problem is too large to be solved via direct factorization. One can resort to iterative approaches, in particular the conjugate gradients method if the matrix is symmetric positive definite. P…

New MCMC method learns sparse preconditioner for high-dimensional problems.

The successive projection algorithm (SPA) can quickly solve a nonnegative matrix factorization problem under a separability assumption. Even if noise is added to the problem, SPA is robust as long as the perturbations caused by the noise are small. In particular, robustness against noise should be high when handling th…

Bias correction improves language model training performance.

New iterative methods improve scalability of Gaussian process approximations for large data.

RL accelerates portfolio optimization and option pricing by dynamically adjusting preconditioner sizes.

New method speeds up kernel-based machine learning for force field reconstruction.

The task of choosing a preconditioner $\boldsymbol{M}$ to use when solving a linear system $\boldsymbol{Ax}=\boldsymbol{b}$ with iterative methods is difficult. For instance, even if one has access to a collection $\boldsymbol{M}_1,\boldsymbol{M}_2,\ldots,\boldsymbol{M}_n$ of candidate preconditioners, it is currently …

New analysis for learning and applying preconditioners in MCMC improves efficiency.

Analyzes learning and applying preconditioners in MCMC for efficiency.

When the design matrix has orthonormal columns, "soft thresholding" the ordinary least squares (OLS) solution produces the Lasso solution [Tibshirani, 1996]. If one uses the Puffer preconditioned Lasso [Jia and Rohe, 2012], then this result generalizes from orthonormal designs to full rank designs (Theorem 1). Theorem …

Despite advances in scalable models, the inference tools used for Gaussian processes (GPs) have yet to fully capitalize on developments in computing hardware. We present an efficient and general approach to GP inference based on Blackbox Matrix-Matrix multiplication (BBMM). BBMM inference uses a modified batched versio…

Nystrom approximation speeds up kernel model training.

We present a novel preconditioning technique for proximal optimization methods that relies on graph algorithms to construct effective preconditioners. Such combinatorial preconditioners arise from partitioning the graph into forests. We prove that certain decompositions lead to a theoretically optimal condition number.…

A new method improves convergence in low-rank approximation.

Stochastic gradient descent (SGD) still is the workhorse for many practical problems. However, it converges slow, and can be difficult to tune. It is possible to precondition SGD to accelerate its convergence remarkably. But many attempts in this direction either aim at solving specialized problems, or result in signif…

Scalable algorithm for sampling Gaussian processes using sparse grids and preconditioners.

New lower bounds show sparse recovery is hard even with multiple preconditioners.

This paper proposes a family of online second order methods for possibly non-convex stochastic optimizations based on the theory of preconditioned stochastic gradient descent (PSGD), which can be regarded as an enhance stochastic Newton method with the ability to handle gradient noise and non-convexity simultaneously. …

A new optimization method reduces memory and compute requirements for deep learning.

Improved convergence for overparameterized low-rank matrix sensing.

Standard gradient descent methods are susceptible to a range of issues that can impede training, such as high correlations and different scaling in parameter space.These difficulties can be addressed by second-order approaches that apply a pre-conditioning matrix to the gradient to improve convergence. Unfortunately, s…

Adaptive methods such as Adam and RMSProp are widely used in deep learning but are not well understood. In this paper, we seek a crisp, clean and precise characterization of their behavior in nonconvex settings. To this end, we first provide a novel view of adaptive methods as preconditioned SGD, where the precondition…

PrecGD restores linear convergence in over-parameterized nonconvex matrix factorization.

New algorithm estimates Gaussian means and covariances efficiently and privately.

Dual Space Preconditioning speeds up gradient descent in overparameterized models.

The paper explores how dynamic preconditioning affects the CLT in online averaging.

MuonEq improves training of matrix-valued parameters by rebalancing momentum before orthogonalization.

Gradient descent converges geometrically to optimal self-attention parameters.

Sketchy reduces memory and compute requirements for adaptive regularization in deep learning.

Unified framework for training neural networks with non-smooth, non-convex regularizers.

We propose two methods for exact Gaussian process (GP) inference and learning on massive image, video, spatial-temporal, or multi-output datasets with missing values (or "gaps") in the observed responses. The first method ignores the gaps using sparse selection matrices and a highly effective low-rank preconditioner is…

Optimizes nonconvex optimization by converting it to static regret minimization.

The successive projection algorithm (SPA) has been known to work well for separable nonnegative matrix factorization (NMF) problems arising in applications, such as topic extraction from documents and endmember detection in hyperspectral images. One of the reasons is in that the algorithm is robust to noise. Gillis and…

ASTRA improves TDA by more accurately approximating iHVP.

The computational and storage complexity of kernel machines presents the primary barrier to their scaling to large, modern, datasets. A common way to tackle the scalability issue is to use the conjugate gradient algorithm, which relieves the constraints on both storage (the kernel matrix need not be stored) and computa…

New algorithm learns LQR with $O(\sqrt{T})$ regret using Langevin dynamics and excitation.

A new method reduces complexity of normalizing flows for MCMC preconditioning.