We study the property of the Fused Lasso Signal Approximator (FLSA) for estimating a blocky signal sequence with additive noise. We transform the FLSA to an ordinary Lasso problem. By studying the property of the design matrix in the transformed Lasso problem, we find that the irrepresentable condition might not hold, …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “preconditioned Lasso”
A note relating ridge regression and OLS p-values to preconditioned sparse penalized regressionstat.ML
When the design matrix has orthonormal columns, "soft thresholding" the ordinary least squares (OLS) solution produces the Lasso solution [Tibshirani, 1996]. If one uses the Puffer preconditioned Lasso [Jia and Rohe, 2012], then this result generalizes from orthonormal designs to full rank designs (Theorem 1). Theorem …
New research shows how preconditioning can solve sparse linear regression problems efficiently.
problem Efficiently solving sparse linear regression problems without restrictive conditions.
method Preconditioned Lasso approach to solve sparse linear regression problems.
result Preconditioning can solve a large class of sparse linear regression problems nearly optimally.
New lower bounds show sparse recovery is hard even with multiple preconditioners.
problem Sparse recovery with ill-conditioned designs is hard for certain algorithms.
method Constructing a single signal distribution that multiple preconditioned Lasso programs fail on.
result Standard sparse random designs are robust to erasures, aiding sparse recovery.
Improved online Lasso reduces regret in sparse linear contextual bandits.
problem Sparse linear contextual bandit problem with inefficient sampling.
method Perturbed adversary approach to alleviate sampling inefficiency.
result Online Lasso achieves regret bound.
SAPPHIRE: Preconditioned Stochastic Variance Reduction for Faster Large-Scale Statistical Learningstat.ML
SAPPHIRE tackles ill-conditioned rERM problems with faster convergence.
problem Ill-conditioned objectives and non-smooth regularizers in rERM.
method Sketch-based preconditioning and scaled proximal mapping.
result Achieves condition-number-free linear convergence.
Layer-wise preconditioning methods improve neural network optimization and feature learning.
problem Suboptimal feature learning in standard optimization algorithms.
method Layer-wise preconditioning methods that introduce preconditioners per axis of each layer's weight tensors.
result Layer-wise preconditioning is necessary for provable feature learning in linear and single-index models.
We provide an online convex optimization algorithm with regret that interpolates between the regret of an algorithm using an optimal preconditioning matrix and one using a diagonal preconditioning matrix. Our regret bound is never worse than that obtained by diagonal preconditioning, and in certain setting even surpass…
Universal preconditioning reduces sequential prediction regret.
problem Improving sequential prediction performance.
method Convolve target sequence with orthogonal polynomial coefficients.
result First sublinear and hidden-dimension-independent regret bounds.
TDprop uses Jacobi preconditioning to improve adaptive optimizers in Deep RL.
problem Improving performance of adaptive optimizers in Deep RL.
method TDprop computes per-parameter learning rates based on Jacobi preconditioning of the TD update rule.
result TDprop matches or exceeds Adam's performance in Deep RL experiments, suggesting Jacobi preconditioning can improve adaptive methods.
Optimal preconditioning improves Langevin sampling efficiency.
problem Improving sampling efficiency in high-dimensional target distributions.
method Optimal preconditioning using Fisher information, applied to MALA.
result Adaptive MCMC scheme significantly outperforms other methods.
Preconditioned neural posterior estimation improves reliability in misspecified models.
problem Reliability issues in neural posterior estimation for misspecified models.
method Preconditioning with data-dependent weights and forest-proximity scores to stabilize and improve accuracy.
result Preconditioned robust neural posterior estimation increases stability and accuracy over standard methods.
Preconditioned gradient methods are among the most general and powerful tools in optimization. However, preconditioning requires storing and manipulating prohibitively large matrices. We describe and analyze a new structure-aware preconditioning algorithm, called Shampoo, for stochastic optimization over tensor spaces.…
In this work, we study data preconditioning, a well-known and long-existing technique, for boosting the convergence of first-order methods for regularized loss minimization. It is well understood that the condition number of the problem, i.e., the ratio of the Lipschitz constant to the strong convexity modulus, has a h…
Bias correction improves language model training performance.
problem Stochastic update bias in preconditioned optimizers.
method Cross-fitted preconditioning and variance-corrected inversion.
result Reduces held-out pretraining loss by 0.15 nats.
Preconditioned non-convex gradient descent improves noisy matrix estimation.
problem Estimating low-rank matrices from noisy measurements.
method Preconditioned non-convex gradient descent for noisy measurements.
result Preconditioned method converges to minimax optimal estimate at a linear rate.
Unified framework for understanding and optimizing training acceleration.
problem Challenges in optimizing training with regularization and acceleration techniques.
method Explains how AdaGrad, RMSProp, and Adam accelerate training, and derives a generalization for -regularization.
result Derives a unified mathematical framework for understanding and optimizing training acceleration.
This paper optimizes diagonal preconditioning to improve matrix condition numbers.
problem Optimizing diagonal preconditioning to reduce matrix condition numbers.
method Reformulated as a quasi-convex problem, solved with bisection and Newton updates.
result Optimal diagonal preconditioners can significantly improve iterative methods.
Two methods solve kernel ridge regression problems efficiently.
problem Solving kernel ridge regression problems with large datasets.
method RPCholesky and KRILL preconditioning techniques.
result Efficient solutions to KRR problems with strong guarantees.
Randomized block-diagonal preconditioning improves parallel learning convergence.
problem Improving convergence of gradient-based optimization methods in parallel settings.
method Randomization of coordinates during optimization to repartition tasks.
result Randomization significantly improves convergence of block-diagonal preconditioned methods.
PolarGrad optimizes deep learning models by considering matrix structure, outperforming Adam and Muon.
problem Efficient optimization of large-scale neural networks and language models.
method A unifying framework for analyzing matrix-aware preconditioned methods, including PolarGrad.
result PolarGrad outperforms Adam and Muon in various tasks.
New method speeds up solving orthogonality constrained problems.
problem Solving orthogonality constrained problems efficiently.
method Riemannian optimization and Riemannian preconditioning.
result Preconditioning improves computational costs and convergence.
In this paper, we analyze different preconditionings designed to enhance robustness of pure-pixel search algorithms, which are used for blind hyperspectral unmixing and which are equivalent to near-separable nonnegative matrix factorization algorithms. Our analysis focuses on the successive projection algorithm (SPA), …
New sampling method using regularized Wasserstein proximal for Gibbs distributions.
problem Sampling from Gibbs distributions with numerical stability and efficiency.
method Preconditioned regularized Wasserstein proximal operator.
result Discrete-time convergence analysis and explicit bias characterization.
A new method improves convergence in low-rank approximation.
problem Efficiently solving large-scale numerical linear algebra problems.
method Error-Powered Sketched Inverse Iteration (EPSI) Method.
result Convergence rate improves at least linearly with sketch size.
Preconditioned NFs speed up sampling from complex posterior distributions in inverse problems.
problem Sampling from posterior distributions of inverse problems with expensive forward operators.
method Preconditioning a conditional normalizing flow (NF) to speed up training.
result Significant speed-ups achieved compared to training NFs from scratch.
A new method reduces complexity of normalizing flows for MCMC preconditioning.
problem Improving sampling efficiency in MCMC algorithms for complex target distributions.
method Factorized preconditioning architecture combining a linear component and a conditional NF.
result Significantly better tail samples and higher effective sample sizes on various distributions.
Adaptive learning rate algorithms such as RMSProp are widely used for training deep neural networks. RMSProp offers efficient training since it uses first order gradients to approximate Hessian-based preconditioning. However, since the first order gradients include noise caused by stochastic optimization, the approxima…
Improved convergence for overparameterized low-rank matrix sensing.
problem Overparameterized low-rank matrix sensing with unknown rank and ill-conditioning.
method ScaledGD() - preconditioned gradient descent method.
result ScaledGD() converges at a constant linear rate after a logarithmic number of iterations.
The paper explores efficient sampling for Bayesian wide neural networks.
problem Sampling from posterior distributions of wide neural networks.
method Preconditioned Crank-Nicolson and Langevin algorithms for reparametrised posterior distributions.
result The preconditioned Crank-Nicolson algorithm improves sampling efficiency in wide networks.
Nonnegative matrix factorization (NMF) under the separability assumption can provably be solved efficiently, even in the presence of noise, and has been shown to be a powerful technique in document classification and hyperspectral unmixing. This problem is referred to as near-separable NMF and requires that there exist…
SignSGD analysis quantifies its effects in high dimensions.
problem Understanding signSGD's effects in high-dimensional settings.
method High-dimensional analysis of signSGD, deriving SDE and ODE for risk.
result Quantification of signSGD's effects: effective learning rate, noise compression, diagonal preconditioning, gradient noise reshaping.
Standard gradient descent methods are susceptible to a range of issues that can impede training, such as high correlations and different scaling in parameter space.These difficulties can be addressed by second-order approaches that apply a pre-conditioning matrix to the gradient to improve convergence. Unfortunately, s…
Stochastic Gradient Langevin Dynamics infuses isotropic gradient noise to SGD to help navigate pathological curvature in the loss landscape for deep networks. Isotropic nature of the noise leads to poor scaling, and adaptive methods based on higher order curvature information such as Fisher Scoring have been proposed t…
Adaptive regularization methods pre-multiply a descent direction by a preconditioning matrix. Due to the large number of parameters of machine learning problems, full-matrix preconditioning methods are prohibitively expensive. We show how to modify full-matrix adaptive regularization in order to make it practical and e…
We propose a novel Riemannian manifold preconditioning approach for the tensor completion problem with rank constraint. A novel Riemannian metric or inner product is proposed that exploits the least-squares structure of the cost function and takes into account the structured symmetry that exists in Tucker decomposition…
In this paper we introduce a parameter dependent class of Krylov-based methods, namely CD, for the solution of symmetric linear systems. We give evidence that in our proposal we generate sequences of conjugate directions, extending some properties of the standard Conjugate Gradient (CG) method, in order to preserve the…
State-of-the-art models are now trained with billions of parameters, reaching hardware limits in terms of memory consumption. This has created a recent demand for memory-efficient optimizers. To this end, we investigate the limits and performance tradeoffs of memory-efficient adaptively preconditioned gradient methods.…
Independent Component Analysis (ICA) is a technique for unsupervised exploration of multi-channel data that is widely used in observational sciences. In its classic form, ICA relies on modeling the data as linear mixtures of non-Gaussian independent sources. The maximization of the corresponding likelihood is a challen…
Gradient-based MCMC for discrete spaces improves sampling performance.
problem Sampling in discrete spaces using traditional methods is challenging.
method Introduced new discrete Metropolis-Hastings samplers inspired by MALA, with a novel preconditioning technique.
result Demonstrated strong empirical performance across various challenging sampling problems.
SLMC improves sampling efficiency for high-dimensional distributions.
problem Sampling from high-dimensional distributions is computationally challenging.
method SLMC projects Langevin updates onto subsampled eigenblocks of a time-varying preconditioner.
result SLMC offers superior adaptability and computational efficiency compared to traditional methods.
Bayesian sparse learning method improves deep neural network efficiency.
problem Sparse learning in deep neural networks with complex geometry.
method Preconditioned stochastic gradient Langevin Dynamics (PSGLD) for sampling and adaptive optimization of hyperparameters.
result The proposed algorithm achieves asymptotic convergence with controlled bias.
Dual Space Preconditioning speeds up gradient descent in overparameterized models.
problem Improving convergence of gradient descent in overparameterized linear models.
method Introducing a novel preconditioner of the form for convex and applying it to overparameterized linear models.
result The iterates of the preconditioned gradient descent converge to a solution satisfying .
This work analyzes Adam's preconditioning effect on quadratic functions and quantifies its impact on condition number.
problem Understanding and quantifying the preconditioning effect of Adam to alleviate ill-conditioning in gradient descent.
method Detailed analysis of Adam's preconditioning effect for quadratic functions, including empirical evidence.
result Adam can mitigate the condition number but at a dimension-dependent cost, with specific bounds for different types of Hessians.
This paper improves linear system solving by optimizing matrix diagonal scaling.
problem Improving the condition number of a matrix for faster iterative methods.
method Left or right diagonal rescaling of the matrix A, with new bounds and algorithms.
result Jacobi preconditioning reduces A's condition number to within a quadratic factor of the best possible scaling.
Polyak-Ruppert CLT for SA-Adam with momentum and non-convergent adaptive preconditioning
problem Adaptive optimizers combining momentum and non-convergent preconditioning
method Proving positive drift stability and a non-autonomous Polyak-Ruppert CLT for SA-Adam
result The iterate-marginal covariance is exactly the plain stochastic gradient descent (SGD) sandwich
Stochastic gradient descent (SGD) still is the workhorse for many practical problems. However, it converges slow, and can be difficult to tune. It is possible to precondition SGD to accelerate its convergence remarkably. But many attempts in this direction either aim at solving specialized problems, or result in signif…
Improved likelihood-free inference using preconditioned neural posterior estimation.
problem Inaccurate posterior estimation in likelihood-free inference methods.
method Preconditioned Neural Posterior Estimation (PNPE) and Sequential PNPE (PSNPE) methods.
result PNPE and PSNPE improve posterior estimation accuracy over NPE and SNPE.