We propose a novel Riemannian manifold preconditioning approach for the tensor completion problem with rank constraint. A novel Riemannian metric or inner product is proposed that exploits the least-squares structure of the cost function and takes into account the structured symmetry that exists in Tucker decomposition…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,181 papers · 148 categories
Trend · papers per month
48 results for “Riemannian preconditioning”
New method speeds up solving orthogonality constrained problems.
problem Solving orthogonality constrained problems efficiently.
method Riemannian optimization and Riemannian preconditioning.
result Preconditioning improves computational costs and convergence.
Optimal preconditioning improves Langevin sampling efficiency.
problem Improving sampling efficiency in high-dimensional target distributions.
method Optimal preconditioning using Fisher information, applied to MALA.
result Adaptive MCMC scheme significantly outperforms other methods.
Stochastic Gradient Descent in Hilbert Scales: Smoothness, Preconditioning and Earlier Stoppingstat.ML
Stochastic Gradient Descent improved for various Hilbert scales and misspecified models.
problem Understanding and optimizing SGD in Hilbert scales for machine learning.
method Extending SGD analysis to Hilbert scales, including Sobolev and Diffusion spaces, and showing the effects of smoothness and preconditioning.
result Violation of smoothness assumption affects learning rate; preconditioning in Hilbert scales reduces the number of iterations for misspecified models.
Improved online learning algorithm with better regret bounds.
problem Online convex optimization with improved regret bounds.
method Matrix-free preconditioning approach.
result Our algorithm outperforms diagonal preconditioning in certain settings.
Layer-wise preconditioning methods improve neural network optimization and feature learning.
problem Suboptimal feature learning in standard optimization algorithms.
method Layer-wise preconditioning methods that introduce preconditioners per axis of each layer's weight tensors.
result Layer-wise preconditioning is necessary for provable feature learning in linear and single-index models.
Adaptive Bayesian sampling technique simplifies mass matrix learning.
problem Complexity in learning mass matrices for adaptive samplers.
method Monte Carlo EM framework with online learning of mass matrices.
result Comparable sampling accuracy to Riemannian samplers but faster.
Universal preconditioning reduces sequential prediction regret.
problem Improving sequential prediction performance.
method Convolve target sequence with orthogonal polynomial coefficients.
result First sublinear and hidden-dimension-independent regret bounds.
Shampoo optimizes tensor spaces with faster convergence.
problem Optimizing models over tensor spaces with large matrices.
method Structure-aware preconditioning for stochastic tensor optimization.
result Shampoo converges faster than existing optimizers.
TDprop uses Jacobi preconditioning to improve adaptive optimizers in Deep RL.
problem Improving performance of adaptive optimizers in Deep RL.
method TDprop computes per-parameter learning rates based on Jacobi preconditioning of the TD update rule.
result TDprop matches or exceeds Adam's performance in Deep RL experiments, suggesting Jacobi preconditioning can improve adaptive methods.
Preconditioned neural posterior estimation improves reliability in misspecified models.
problem Reliability issues in neural posterior estimation for misspecified models.
method Preconditioning with data-dependent weights and forest-proximity scores to stabilize and improve accuracy.
result Preconditioned robust neural posterior estimation increases stability and accuracy over standard methods.
In this work, we study data preconditioning, a well-known and long-existing technique, for boosting the convergence of first-order methods for regularized loss minimization. It is well understood that the condition number of the problem, i.e., the ratio of the Lipschitz constant to the strong convexity modulus, has a h…
Adaptively preconditions SGLD for faster convergence and better generalization.
problem Pathological curvature in deep network loss landscapes.
method Adaptive estimation of noise parameters to precondition isotropic gradient noise.
result Adaptively preconditioned SGLD achieves faster convergence and generalization equivalent of SGD.
Bias correction improves language model training performance.
problem Stochastic update bias in preconditioned optimizers.
method Cross-fitted preconditioning and variance-corrected inversion.
result Reduces held-out pretraining loss by 0.15 nats.
Preconditioned non-convex gradient descent improves noisy matrix estimation.
problem Estimating low-rank matrices from noisy measurements.
method Preconditioned non-convex gradient descent for noisy measurements.
result Preconditioned method converges to minimax optimal estimate at a linear rate.
Unified framework for understanding and optimizing training acceleration.
problem Challenges in optimizing training with regularization and acceleration techniques.
method Explains how AdaGrad, RMSProp, and Adam accelerate training, and derives a generalization for -regularization.
result Derives a unified mathematical framework for understanding and optimizing training acceleration.
Efficiently preconditions machine learning problems with adaptive regularization.
problem Prohibitively expensive full-matrix adaptive regularization for large parameter problems.
method Modified full-matrix adaptive regularization with efficient inverse square root computation.
result Improved convergence rates and better solutions through careful preconditioning.
This paper optimizes diagonal preconditioning to improve matrix condition numbers.
problem Optimizing diagonal preconditioning to reduce matrix condition numbers.
method Reformulated as a quasi-convex problem, solved with bisection and Newton updates.
result Optimal diagonal preconditioners can significantly improve iterative methods.
Two methods solve kernel ridge regression problems efficiently.
problem Solving kernel ridge regression problems with large datasets.
method RPCholesky and KRILL preconditioning techniques.
result Efficient solutions to KRR problems with strong guarantees.
Randomized block-diagonal preconditioning improves parallel learning convergence.
problem Improving convergence of gradient-based optimization methods in parallel settings.
method Randomization of coordinates during optimization to repartition tasks.
result Randomization significantly improves convergence of block-diagonal preconditioned methods.
PolarGrad optimizes deep learning models by considering matrix structure, outperforming Adam and Muon.
problem Efficient optimization of large-scale neural networks and language models.
method A unifying framework for analyzing matrix-aware preconditioned methods, including PolarGrad.
result PolarGrad outperforms Adam and Muon in various tasks.
In this paper, we analyze different preconditionings designed to enhance robustness of pure-pixel search algorithms, which are used for blind hyperspectral unmixing and which are equivalent to near-separable nonnegative matrix factorization algorithms. Our analysis focuses on the successive projection algorithm (SPA), …
New sampling method using regularized Wasserstein proximal for Gibbs distributions.
problem Sampling from Gibbs distributions with numerical stability and efficiency.
method Preconditioned regularized Wasserstein proximal operator.
result Discrete-time convergence analysis and explicit bias characterization.
A new method improves convergence in low-rank approximation.
problem Efficiently solving large-scale numerical linear algebra problems.
method Error-Powered Sketched Inverse Iteration (EPSI) Method.
result Convergence rate improves at least linearly with sketch size.
Preconditioned NFs speed up sampling from complex posterior distributions in inverse problems.
problem Sampling from posterior distributions of inverse problems with expensive forward operators.
method Preconditioning a conditional normalizing flow (NF) to speed up training.
result Significant speed-ups achieved compared to training NFs from scratch.
New research shows how preconditioning can solve sparse linear regression problems efficiently.
problem Efficiently solving sparse linear regression problems without restrictive conditions.
method Preconditioned Lasso approach to solve sparse linear regression problems.
result Preconditioning can solve a large class of sparse linear regression problems nearly optimally.
FOP improves deep learning optimizers with minimal computational overhead.
problem Training deep learning models can be hindered by high correlations and different scaling in parameter space.
method FOP uses first-order information to learn a preconditioning matrix that improves convergence without the high computational cost of second-order methods.
result FOP improves performance of standard deep learning optimizers on visual classification and reinforcement learning tasks.
A new method reduces memory usage for large models by three orders of magnitude.
problem Memory limitations in training large models.
method Extreme tensoring for adaptive preconditioning.
result Significant reduction in memory usage without performance degradation.
A new method reduces complexity of normalizing flows for MCMC preconditioning.
problem Improving sampling efficiency in MCMC algorithms for complex target distributions.
method Factorized preconditioning architecture combining a linear component and a conditional NF.
result Significantly better tail samples and higher effective sample sizes on various distributions.
Adaptive learning rate algorithms such as RMSProp are widely used for training deep neural networks. RMSProp offers efficient training since it uses first order gradients to approximate Hessian-based preconditioning. However, since the first order gradients include noise caused by stochastic optimization, the approxima…
Improved convergence for overparameterized low-rank matrix sensing.
problem Overparameterized low-rank matrix sensing with unknown rank and ill-conditioning.
method ScaledGD() - preconditioned gradient descent method.
result ScaledGD() converges at a constant linear rate after a logarithmic number of iterations.
The paper explores efficient sampling for Bayesian wide neural networks.
problem Sampling from posterior distributions of wide neural networks.
method Preconditioned Crank-Nicolson and Langevin algorithms for reparametrised posterior distributions.
result The preconditioned Crank-Nicolson algorithm improves sampling efficiency in wide networks.
Nonnegative matrix factorization (NMF) under the separability assumption can provably be solved efficiently, even in the presence of noise, and has been shown to be a powerful technique in document classification and hyperspectral unmixing. This problem is referred to as near-separable NMF and requires that there exist…
SignSGD analysis quantifies its effects in high dimensions.
problem Understanding signSGD's effects in high-dimensional settings.
method High-dimensional analysis of signSGD, deriving SDE and ODE for risk.
result Quantification of signSGD's effects: effective learning rate, noise compression, diagonal preconditioning, gradient noise reshaping.
New graph-based preconditioners speed up optimization on graph data.
problem Optimizing functions on graph data structures.
method Construct combinatorial preconditioners from graph partitions.
result Provable optimal condition numbers for certain decompositions.
A note relating ridge regression and OLS p-values to preconditioned sparse penalized regressionstat.ML
When the design matrix has orthonormal columns, "soft thresholding" the ordinary least squares (OLS) solution produces the Lasso solution [Tibshirani, 1996]. If one uses the Puffer preconditioned Lasso [Jia and Rohe, 2012], then this result generalizes from orthonormal designs to full rank designs (Theorem 1). Theorem …
In this paper we introduce a parameter dependent class of Krylov-based methods, namely CD, for the solution of symmetric linear systems. We give evidence that in our proposal we generate sequences of conjugate directions, extending some properties of the standard Conjugate Gradient (CG) method, in order to preserve the…
Independent Component Analysis (ICA) is a technique for unsupervised exploration of multi-channel data that is widely used in observational sciences. In its classic form, ICA relies on modeling the data as linear mixtures of non-Gaussian independent sources. The maximization of the corresponding likelihood is a challen…
Gradient-based MCMC for discrete spaces improves sampling performance.
problem Sampling in discrete spaces using traditional methods is challenging.
method Introduced new discrete Metropolis-Hastings samplers inspired by MALA, with a novel preconditioning technique.
result Demonstrated strong empirical performance across various challenging sampling problems.
We study the property of the Fused Lasso Signal Approximator (FLSA) for estimating a blocky signal sequence with additive noise. We transform the FLSA to an ordinary Lasso problem. By studying the property of the design matrix in the transformed Lasso problem, we find that the irrepresentable condition might not hold, …
SLMC improves sampling efficiency for high-dimensional distributions.
problem Sampling from high-dimensional distributions is computationally challenging.
method SLMC projects Langevin updates onto subsampled eigenblocks of a time-varying preconditioner.
result SLMC offers superior adaptability and computational efficiency compared to traditional methods.
Study shows how algorithmic choices affect optimal batch sizes in neural networks.
problem Understanding how batch size impacts neural network training efficiency.
method Experiments and analysis of a simple quadratic model to study algorithmic choices.
result Preconditioned optimizers like Adam and K-FAC allow larger batch sizes before diminishing returns.
Dual Space Preconditioning speeds up gradient descent in overparameterized models.
problem Improving convergence of gradient descent in overparameterized linear models.
method Introducing a novel preconditioner of the form for convex and applying it to overparameterized linear models.
result The iterates of the preconditioned gradient descent converge to a solution satisfying .
Bayesian sparse learning method improves deep neural network efficiency.
problem Sparse learning in deep neural networks with complex geometry.
method Preconditioned stochastic gradient Langevin Dynamics (PSGLD) for sampling and adaptive optimization of hyperparameters.
result The proposed algorithm achieves asymptotic convergence with controlled bias.
This work analyzes Adam's preconditioning effect on quadratic functions and quantifies its impact on condition number.
problem Understanding and quantifying the preconditioning effect of Adam to alleviate ill-conditioning in gradient descent.
method Detailed analysis of Adam's preconditioning effect for quadratic functions, including empirical evidence.
result Adam can mitigate the condition number but at a dimension-dependent cost, with specific bounds for different types of Hessians.
This paper improves linear system solving by optimizing matrix diagonal scaling.
problem Improving the condition number of a matrix for faster iterative methods.
method Left or right diagonal rescaling of the matrix A, with new bounds and algorithms.
result Jacobi preconditioning reduces A's condition number to within a quadratic factor of the best possible scaling.
New method efficiently learns positive-definite curvature for neural nets.
problem Efficiently learn positive-definite curvature for neural net training.
method Spectral-factorized positive-definite curvature learning approach.
result Efficiently applies arbitrary matrix roots and generic curvature learning.
Polyak-Ruppert CLT for SA-Adam with momentum and non-convergent adaptive preconditioning
problem Adaptive optimizers combining momentum and non-convergent preconditioning
method Proving positive drift stability and a non-autonomous Polyak-Ruppert CLT for SA-Adam
result The iterate-marginal covariance is exactly the plain stochastic gradient descent (SGD) sandwich