PSGD accelerates RNN training, achieving competitive performance.
problem Training recurrent neural networks, especially those with long-term memory requirements.
method Preconditioned stochastic gradient descent (PSGD) algorithm.
result PSGD achieves highly competitive performance on RNN training tasks.
Stochastic Gradient Descent improved for various Hilbert scales and misspecified models.
problem Understanding and optimizing SGD in Hilbert scales for machine learning.
method Extending SGD analysis to Hilbert scales, including Sobolev and Diffusion spaces, and showing the effects of smoothness and preconditioning.
result Violation of smoothness assumption affects learning rate; preconditioning in Hilbert scales reduces the number of iterations for misspecified models.
New approach for tensor completion using Riemannian manifold optimization.
problem Tensor completion with rank constraint.
method Riemannian manifold preconditioning approach with novel metric.
result Robust algorithms outperform state-of-the-art across various datasets.
Uncertainty sampling is explained as a gradient step on a smoothed loss, leading to better parameters.
problem Reducing the amount of data required to learn a classifier.
method Interprets uncertainty sampling as a preconditioned stochastic gradient step on a smoothed zero-one loss.
result Uncertainty sampling converges to stationary points of the smoothed population zero-one loss.
Stochastic gradient descent (SGD) still is the workhorse for many practical problems. However, it converges slow, and can be difficult to tune. It is possible to precondition SGD to accelerate its convergence remarkably. But many attempts in this direction either aim at solving specialized problems, or result in signif…
Study shows how algorithmic choices affect optimal batch sizes in neural networks.
problem Understanding how batch size impacts neural network training efficiency.
method Experiments and analysis of a simple quadratic model to study algorithmic choices.
result Preconditioned optimizers like Adam and K-FAC allow larger batch sizes before diminishing returns.
Bias correction improves language model training performance.
problem Stochastic update bias in preconditioned optimizers.
method Cross-fitted preconditioning and variance-corrected inversion.
result Reduces held-out pretraining loss by 0.15 nats.
Preconditioned non-convex gradient descent improves noisy matrix estimation.
problem Estimating low-rank matrices from noisy measurements.
method Preconditioned non-convex gradient descent for noisy measurements.
result Preconditioned method converges to minimax optimal estimate at a linear rate.
New theory explains why normalization is preferred in SGD under heavy-tailed noise.
problem Understanding why normalization is preferred in stochastic gradient descent (SGD) under heavy-tailed noise.
method Developed a worst-case complexity theory for stochastically preconditioned SGD and its variants.
result Normalization guarantees convergence at optimal rates, while clipping may fail in the worst case.
Effective training of deep neural networks suffers from two main issues. The first is that the parameter spaces of these models exhibit pathological curvature. Recent methods address this problem by using adaptive preconditioning for Stochastic Gradient Descent (SGD). These methods improve convergence by adapting to th…
Stochastic gradient descent improves Gaussian process regression.
problem Efficiently solving large linear systems in Gaussian process regression.
method Developed a stochastic dual descent algorithm using insights from optimisation and kernel communities.
result Stochastic gradient descent is highly effective when done right.
Paper introduces online second order methods for non-convex stochastic optimization.
problem Non-convex stochastic optimization problems.
method Enhanced preconditioned stochastic gradient descent (PSGD) with improved implementations.
result Demonstrates PSGD's advantages in generalization and convergence speed.
FOP improves deep learning optimizers with minimal computational overhead.
problem Training deep learning models can be hindered by high correlations and different scaling in parameter space.
method FOP uses first-order information to learn a preconditioning matrix that improves convergence without the high computational cost of second-order methods.
result FOP improves performance of standard deep learning optimizers on visual classification and reinforcement learning tasks.
SGLD improves sampling for Bayesian models on large datasets.
problem Poor convergence and mixing times in SGLD for models with scale and correlation variations.
method Preconditioning noise vector, comparing normalization approaches.
result Preconditioning improves mixing times and regularization effects.
New analysis shows GMD can converge linearly under PL-like conditions.
problem Establishing linear convergence for generalized mirror descent.
method PL-based analysis for time-dependent mirrors, Taylor-series approach for stochastic GMD.
result Linear convergence of stochastic GMD under PL-like conditions.
Enhances SVGD with matrix-valued kernels for faster inference.
problem Efficient approximate inference in complex probability landscapes.
method Integrates geometric information through matrix-valued kernels in SVGD.
result Significant improvement in real-world Bayesian inference tasks.
New analysis shows SNG's effectiveness in small samples.
problem Limited understanding of SNG in small data settings.
method Sketch-and-project analysis of SNG.
result Global convergence and rate characterization for SNG.
Dual Space Preconditioning speeds up gradient descent in overparameterized models.
problem Improving convergence of gradient descent in overparameterized linear models.
method Introducing a novel preconditioner of the form ablaK for convex K and applying it to overparameterized linear models. result The iterates of the preconditioned gradient descent converge to a solution W∞ satisfying XW∞=Y. Improved convergence for overparameterized low-rank matrix sensing.
problem Overparameterized low-rank matrix sensing with unknown rank and ill-conditioning.
method ScaledGD(λ) - preconditioned gradient descent method. result ScaledGD(λ) converges at a constant linear rate after a logarithmic number of iterations. The paper studies efficient Hessian fitting methods for stochastic optimization.
problem Efficient Hessian fitting for stochastic optimization.
method Preconditioned Stochastic Gradient Descent (PSGD) method and Lie groups.
result Hessian fitting problem is strongly convex in certain Lie groups.
Improved SGLD for neural networks using Fisher matrix approximations.
problem Overfitting in neural networks.
method Natural Langevin dynamics with Fisher matrix preconditioning.
result Fisher matrix preconditioning improves SGLD for large neural networks.
Gradient descent with preconditioning finds global optima in overparameterized nonconvex factorization.
problem Finding global optima in nonconvex Burer-Monteiro factorization.
method Preconditioned gradient descent for overparameterized nonconvex function minimization.
result Gradient descent with preconditioning achieves linear convergence in the overparameterized case.
Adaptively preconditions SGLD for faster convergence and better generalization.
problem Pathological curvature in deep network loss landscapes.
method Adaptive estimation of noise parameters to precondition isotropic gradient noise.
result Adaptively preconditioned SGLD achieves faster convergence and generalization equivalent of SGD.
Polyak-Ruppert CLT for SA-Adam with momentum and non-convergent adaptive preconditioning
problem Adaptive optimizers combining momentum and non-convergent preconditioning
method Proving positive drift stability and a non-autonomous Polyak-Ruppert CLT for SA-Adam
result The iterate-marginal covariance is exactly the plain stochastic gradient descent (SGD) sandwich
WarpGrad efficiently learns preconditioning matrices for gradient descent across task distributions.
problem Learning efficient update rules for rapid new task learning.
method Interleaves warp-layers between task-learner layers to meta-learn preconditioning matrices.
result WarpGrad scales to large meta-learning problems and improves across various learning settings.
Shampoo optimizes tensor spaces with faster convergence.
problem Optimizing models over tensor spaces with large matrices.
method Structure-aware preconditioning for stochastic tensor optimization.
result Shampoo converges faster than existing optimizers.
New method speeds up training of large kernel models.
problem Scaling kernel machines to large datasets and model sizes.
method Delayed projections in Preconditioned Stochastic Gradient Descent (PSGD).
result Significant training speed up over existing methods.
Gradient descent converges geometrically to optimal self-attention parameters.
problem Training softmax self-attention layers for linear regression.
method Structure-aware gradient descent with preconditioner and regularizer.
result Gradient descent converges geometrically to global minima.
New stability analysis improves generalization of multipass SGD.
problem Improper preconditioning affects generalization in multipass SGD.
method Developed on-average stability analysis for multipass SGD.
result Proper preconditioning yields optimal effective dimension dependence.
APGD algorithm efficiently recovers over-parameterized matrices from noisy measurements.
problem Matrix sensing problem with over-parameterization and noisy measurements.
method Alternating preconditioned gradient descent (APGD) algorithm incorporating preconditioning terms.
result APGD converges to a near-optimal error at a linear rate.
Improved online learning algorithm with better regret bounds.
problem Online convex optimization with improved regret bounds.
method Matrix-free preconditioning approach.
result Our algorithm outperforms diagonal preconditioning in certain settings.
Higher-order ODE solvers improve deep learning performance.
problem Improving deep learning performance using higher-order ODE solvers.
method Evaluation and improvement of Runge-Kutta (RK) methods for deep learning.
result Higher-order RK solvers can improve deep learning performance by incorporating key ingredients of optimizers.
Preconditioned conjugate gradients improve kernel machine scalability.
problem Poor convergence of conjugate gradients on kernel matrices.
method Developed preconditioned conjugate gradients for kernel machines.
result Preconditioned conjugate gradients outperform state-of-the-art approximations.
Bayesian sparse learning method improves deep neural network efficiency.
problem Sparse learning in deep neural networks with complex geometry.
method Preconditioned stochastic gradient Langevin Dynamics (PSGLD) for sampling and adaptive optimization of hyperparameters.
result The proposed algorithm achieves asymptotic convergence with controlled bias.
We reformulate linear systems into stochastic problems for faster solutions.
problem Efficiently solving linear systems with stochastic methods.
method Developed a family of reformulations into stochastic problems, analyzed algorithms with global linear convergence rates.
result Found parameters leading to a sufficiently small condition number for faster solutions.
SDProp improves deep neural network training efficiency by noise handling.
problem Inaccurate learning rate approximation in adaptive algorithms like RMSProp.
method SDProp uses covariance matrix preconditioning to handle noise in first order gradients.
result SDProp outperforms RMSProp and variants in various neural networks.
New method speeds up distributed linear regression.
problem Efficiently solve distributed linear regression problems.
method Iteratively Pre-conditioned Stochastic Gradient Descent (IPSG)
result Converges linearly in expectation to the solution.
Nystrom approximation speeds up kernel model training.
problem Slow convergence in kernel models due to poor conditioning.
method Spectral preconditioning with Nystrom approximation for scalability.
result Nystrom approximation accelerates gradient descent nearly as well as exact preconditioner.
Unified approach to adaptive regularization in online and stochastic optimization.
problem Improving convergence in stochastic optimization by adjusting gradient geometry.
method Develops a framework to analyze and derive adaptive online optimization algorithms.
result Simpler convergence proofs for existing methods like AdaGrad and Online Newton Step.
Disputes the empirical Fisher approximation for natural gradient descent.
problem The empirical Fisher approximation fails to capture second-order information in general.
method Comparison of empirical Fisher and Fisher information matrices.
result The empirical Fisher does not generally approximate the Fisher or Hessian.
PrecGD restores linear convergence in over-parameterized nonconvex matrix factorization.
problem Slow convergence of local search algorithms in over-parameterized nonconvex matrix factorization.
method Preconditioned Gradient Descent (PrecGD) with an inexpensive ℓ2 regularization. result PrecGD restores linear convergence rate even in the over-parameterized case.
APO optimizes neural network parameters by amortizing proximal point methods.
problem Optimizing neural network parameters online and adaptively.
method APO framework that meta-learns proximal point parameters.
result APO can recover and outperform existing optimizers and schedules.
Unified framework for two types of matrix Lie group preconditioners in SGD.
problem Improving optimization efficiency in machine learning models.
method Unified framework for Newton and Fisher type preconditioners on matrix Lie groups.
result Efficient estimation of preconditioners on matrix Lie groups.
NLCG optimizes DNN training, especially with large mini-batches.
problem Improving convergence speed in large-scale DNN training.
method Stochastic Preconditioned Nonlinear Conjugate Gradient (SP-NLCG) algorithm.
result NLCG improves DNN training accuracy by over 10 percentage points at large mini-batch sizes.
New method improves MMD estimation without convexity assumptions.
problem Lack of theoretical guarantees for MMD estimation algorithms.
method Preconditioned gradient descent (PGD) scheme for MMD optimization.
result PGD scheme converges globally under specific conditions.
This work analyzes Adam's preconditioning effect on quadratic functions and quantifies its impact on condition number.
problem Understanding and quantifying the preconditioning effect of Adam to alleviate ill-conditioning in gradient descent.
method Detailed analysis of Adam's preconditioning effect for quadratic functions, including empirical evidence.
result Adam can mitigate the condition number but at a dimension-dependent cost, with specific bounds for different types of Hessians.
SAPPHIRE tackles ill-conditioned rERM problems with faster convergence.
problem Ill-conditioned objectives and non-smooth regularizers in rERM.
method Sketch-based preconditioning and scaled proximal mapping.
result Achieves condition-number-free linear convergence.
ScaledGD accelerates ill-conditioned low-rank estimation.
problem Slow convergence of gradient descent in ill-conditioned problems.
method Scaled gradient descent (ScaledGD) with preconditioning.
result Linear convergence rate independent of condition number.