arXiv research

AlgoPerf competition evaluates neural network training speed-ups.

This work analyzes Adam's preconditioning effect on quadratic functions and quantifies its impact on condition number.

Most known four-dimensional cohomogeneity-one Einstein metrics are diagonal in the basis defined by the left-invariant one-forms, though some essentially non-diagonal ones are known. We consider the problem of explicitly seeking non-diagonal Einstein metrics, and we find solutions which in some cases exhaust the possib…

New method efficiently learns positive-definite curvature for neural nets.

Study on stability of non-diagonal Einstein metrics on specific homogeneous spaces.

We study Lorentz hypersurfaces $M_{1}^{n}$ in $E_{1}^{n+1}$ satisfying $\triangle \vec {H}= α\vec {H}$ with non diagonal shape operator, having complex eigenvalues. We prove that every such Lorentz hypersurface in $E_{1}^{n+1}$ having at most five distinct principal curvatures has constant mean curvature.

We study the existence of invariant Einstein metrics on real flag manifolds associated to simple and non-compact split real forms of complex classical Lie algebras whose isotropy representation decomposes into two or three irreducible sub-representations. In this situation, one can have equivalent sub-modules, leading …

Layer-wise preconditioning methods improve neural network optimization and feature learning.

We provide an online convex optimization algorithm with regret that interpolates between the regret of an algorithm using an optimal preconditioning matrix and one using a diagonal preconditioning matrix. Our regret bound is never worse than that obtained by diagonal preconditioning, and in certain setting even surpass…

Universal preconditioning reduces sequential prediction regret.

TDprop uses Jacobi preconditioning to improve adaptive optimizers in Deep RL.

Optimal preconditioning improves Langevin sampling efficiency.

Preconditioned neural posterior estimation improves reliability in misspecified models.

We study the anti-self-dual equation for non-diagonal SU(2)-invariant metrics and give an equivalent ninth-order system. This system reduce to a sixth-order system if the metric is in the conformal class of scalar-flat-Kaehler metric.

Preconditioned gradient methods are among the most general and powerful tools in optimization. However, preconditioning requires storing and manipulating prohibitively large matrices. We describe and analyze a new structure-aware preconditioning algorithm, called Shampoo, for stochastic optimization over tensor spaces.…

In this work, we study data preconditioning, a well-known and long-existing technique, for boosting the convergence of first-order methods for regularized loss minimization. It is well understood that the condition number of the problem, i.e., the ratio of the Lipschitz constant to the strong convexity modulus, has a h…

Bias correction improves language model training performance.

Preconditioned non-convex gradient descent improves noisy matrix estimation.

Unified framework for understanding and optimizing training acceleration.

This paper optimizes diagonal preconditioning to improve matrix condition numbers.

Two methods solve kernel ridge regression problems efficiently.

Randomized block-diagonal preconditioning improves parallel learning convergence.

PolarGrad optimizes deep learning models by considering matrix structure, outperforming Adam and Muon.

New method speeds up solving orthogonality constrained problems.

In this paper, we analyze different preconditionings designed to enhance robustness of pure-pixel search algorithms, which are used for blind hyperspectral unmixing and which are equivalent to near-separable nonnegative matrix factorization algorithms. Our analysis focuses on the successive projection algorithm (SPA), …

New sampling method using regularized Wasserstein proximal for Gibbs distributions.

A new method improves convergence in low-rank approximation.

Preconditioned NFs speed up sampling from complex posterior distributions in inverse problems.

New research shows how preconditioning can solve sparse linear regression problems efficiently.

Study left-invariant Codazzi tensors and harmonic curvature on Lorentzian Lie groups.

A new method reduces complexity of normalizing flows for MCMC preconditioning.

Adaptive learning rate algorithms such as RMSProp are widely used for training deep neural networks. RMSProp offers efficient training since it uses first order gradients to approximate Hessian-based preconditioning. However, since the first order gradients include noise caused by stochastic optimization, the approxima…

Improved convergence for overparameterized low-rank matrix sensing.

The paper explores efficient sampling for Bayesian wide neural networks.

Nonnegative matrix factorization (NMF) under the separability assumption can provably be solved efficiently, even in the presence of noise, and has been shown to be a powerful technique in document classification and hyperspectral unmixing. This problem is referred to as near-separable NMF and requires that there exist…

SignSGD analysis quantifies its effects in high dimensions.

When the design matrix has orthonormal columns, "soft thresholding" the ordinary least squares (OLS) solution produces the Lasso solution [Tibshirani, 1996]. If one uses the Puffer preconditioned Lasso [Jia and Rohe, 2012], then this result generalizes from orthonormal designs to full rank designs (Theorem 1). Theorem …

Standard gradient descent methods are susceptible to a range of issues that can impede training, such as high correlations and different scaling in parameter space.These difficulties can be addressed by second-order approaches that apply a pre-conditioning matrix to the gradient to improve convergence. Unfortunately, s…

Stochastic Gradient Langevin Dynamics infuses isotropic gradient noise to SGD to help navigate pathological curvature in the loss landscape for deep networks. Isotropic nature of the noise leads to poor scaling, and adaptive methods based on higher order curvature information such as Fisher Scoring have been proposed t…

Adaptive regularization methods pre-multiply a descent direction by a preconditioning matrix. Due to the large number of parameters of machine learning problems, full-matrix preconditioning methods are prohibitively expensive. We show how to modify full-matrix adaptive regularization in order to make it practical and e…

We propose a novel Riemannian manifold preconditioning approach for the tensor completion problem with rank constraint. A novel Riemannian metric or inner product is proposed that exploits the least-squares structure of the cost function and takes into account the structured symmetry that exists in Tucker decomposition…

In this paper we introduce a parameter dependent class of Krylov-based methods, namely CD, for the solution of symmetric linear systems. We give evidence that in our proposal we generate sequences of conjugate directions, extending some properties of the standard Conjugate Gradient (CG) method, in order to preserve the…

State-of-the-art models are now trained with billions of parameters, reaching hardware limits in terms of memory consumption. This has created a recent demand for memory-efficient optimizers. To this end, we investigate the limits and performance tradeoffs of memory-efficient adaptively preconditioned gradient methods.…

Independent Component Analysis (ICA) is a technique for unsupervised exploration of multi-channel data that is widely used in observational sciences. In its classic form, ICA relies on modeling the data as linear mixtures of non-Gaussian independent sources. The maximization of the corresponding likelihood is a challen…

Gradient-based MCMC for discrete spaces improves sampling performance.

We study the property of the Fused Lasso Signal Approximator (FLSA) for estimating a blocky signal sequence with additive noise. We transform the FLSA to an ordinary Lasso problem. By studying the property of the design matrix in the transformed Lasso problem, we find that the irrepresentable condition might not hold, …

SLMC improves sampling efficiency for high-dimensional distributions.

Bayesian sparse learning method improves deep neural network efficiency.