A new method improves stochastic gradient descent for faster and more efficient estimation.
problem Efficient and fast parametric estimation methods.
method Projected stochastic gradient descent corrected by Fisher scoring.
result The method is faster and more efficient than traditional methods.
A new method for Bayesian inference in high dimensions using projected Stein variational gradient descent.
problem Bayesian inference challenges in high-dimensional data.
method Adapting Stein variational gradient descent to exploit intrinsic low dimensionality of data.
result pSVGD is more accurate and efficient than SVGD, especially in high-dimensional settings.
Gradient descent at edge of stability stabilizes implicitly, following projected gradient descent.
problem Gradient descent's stability and sharpness behavior at the edge of instability.
method Cubic Taylor expansion analysis of gradient descent dynamics.
result Gradient descent at edge of stability implicitly follows projected gradient descent.
A new method for Bayesian inference tackles high-dimensional problems.
problem Bayesian inference in high-dimensional settings with kernel density estimation issues.
method Projected Wasserstein gradient descent (pWGD) method to overcome curse of dimensionality.
result pWGD method effectively addresses high-dimensional Bayesian inference problems.
We study alignment in linear neural networks and its relation to gradient descent.
problem Understanding alignment in linear neural networks and its impact on training.
method Defined alignment for fully connected networks, analyzed alignment under gradient descent, and compared gradient descent to projected gradient descent for layer-constrained networks.
result Gradient descent can converge linearly to a global minimum when alignment is invariant, and alignment is impossible with large datasets in layer-constrained networks.
We study the projected gradient descent method on low-rank matrix problems with a strongly convex objective. We use the Burer-Monteiro factorization approach to implicitly enforce low-rankness; such factorization introduces non-convexity in the objective. We focus on constraint sets that include both positive semi-defi…
Three new efficient algorithms project vectors onto weighted l1 ball.
problem Sparse system identification and feature selection.
method Projected gradient descent algorithms with linear or highly competitive quadratic worst case complexities.
result Efficient tools for machine learning methods like compress sensing and feature selection.
Paper studies PSGD for constrained optimization problems and its statistical properties.
problem Online inference for constrained optimization problems.
method Stochastic gradient descent with projection (PSGD) for constrained optimization.
result Limiting distribution of PSGD-based estimates under linear-equality constraints.
In this work we introduce a conditional accelerated lazy stochastic gradient descent algorithm with optimal number of calls to a stochastic first-order oracle and convergence rate O(ε21) improving over the projection-free, Online Frank-Wolfe based stochastic gradient descent of Hazan an…
In this paper, we consider the problem of learning high-dimensional tensor regression problems with low-rank structure. One of the core challenges associated with learning high-dimensional models is computation since the underlying optimization problems are often non-convex. While convex relaxations could lead to polyn…
We propose a projected semi-stochastic gradient descent method with mini-batch for improving both the theoretical complexity and practical performance of the general stochastic gradient descent method (SGD). We are able to prove linear convergence under weak strong convexity assumption. This requires no strong convexit…
The paper interprets diffusion models as gradient descent and proposes a new sampler.
problem Improving the efficiency and quality of diffusion models.
method Interprets diffusion models as gradient descent and proposes a new sampler.
result The new sampler achieves state-of-the-art FID scores and generates high quality samples.
A new algorithm estimates sparse gradients on graphs with improved risk bounds.
problem Estimating sparse gradients on graph-structured data.
method Tree-Projected Gradient Descent algorithm for gradient-sparse parameters.
result Achieves risk bound of ns∗log(1+s∗p). In this paper, we aim at providing an introduction to the gradient descent based optimization algorithms for learning deep neural network models. Deep learning models involving multiple nonlinear projection layers are very challenging to train. Nowadays, most of the deep learning model training still relies on the back…
PGD-trained models have a preferential direction in their gradients, which improves robustness.
problem Mathematical lack of clarity in the direction of preferential gradient alignment after adversarial training.
method Proposed a novel definition of preferential direction and evaluated it using a metric based on GANs.
result PGD-trained models have higher alignment with the proposed preferential direction than baseline models.
Faster reconstruction of compressed signals using conditional GAN and NPGD.
problem Recovering compressed signals from measurements.
method Network-based projected gradient descent (NPGD) combined with measurement-conditional generative adversarial networks (GANs/BEGANs).
result Significant speed-up in reconstruction (up to 140-175 times faster).
FP uses random projections to train networks without feedback, achieving comparable performance to backpropagation.
problem Training neural networks without feedback from downstream layers.
method Forward Projection (FP) method that uses randomised nonlinear projections and closed-form regression.
result FP achieves comparable generalisation to backpropagation methods with a single forward pass, offering significant speedup.
A parameter-free PGD algorithm for convex optimization.
problem Minimizing convex functions over convex sets.
method A fully adaptive AdaGrad variant of PGD without parameters or restarts.
result Optimal convergence rates for cumulative regret.
Stochastic approximation algorithms show exponential progress bounds.
problem Analyzing the convergence of stochastic approximation algorithms.
method Developed geometric ergodicity proofs to establish exponential concentration bounds.
result Proved faster convergence rates for specific algorithms.
New method speeds up PGD for CV robustness evaluation.
problem Computational inefficiency of PGD for robustness evaluation.
method Early termination of PGD based on cycle detection.
result Large speedup factors without sacrificing robustness.
Develops first and second-order pseudo-mirror descent methods for nonnegative function estimation.
problem Nonnegative function estimation in settings like MLE and trajectory optimization.
method First and second-order pseudo-mirror descent with pseudo-gradients and projections.
result Establishes tradeoffs and non-asymptotic bounds on model complexity.
Accelerated optimization methods improve robustness and privacy in estimation.
problem Improving robustness and privacy in estimation methods.
method Accelerated gradient methods based on Frank-Wolfe and projected gradient descent, with tailored learning rates and Nesterov's momentum.
result Reduction in iteration complexity, leading to stronger statistical guarantees.
A new method for SVGD reduces variance in high dimensions.
problem High-dimensional variance in SVGD.
method Grassmann Stein Variational Gradient Descent (GSVGD) projects onto arbitrary subspaces and uses coupled Grassmann-valued diffusion.
result GSVGD explores high-dimensional problems with intrinsic low-dimensional structure efficiently.
New method improves MMD estimation without convexity assumptions.
problem Lack of theoretical guarantees for MMD estimation algorithms.
method Preconditioned gradient descent (PGD) scheme for MMD optimization.
result PGD scheme converges globally under specific conditions.
SPGD improves adversarial training efficiency and accuracy.
problem Improving adversarial training efficiency and accuracy with fewer steps.
method Adversarial-sample generation from a frequency domain perspective, extending PGD to the frequency domain.
result SPGD achieves greater adversarial accuracy compared to PGD with fewer attack steps.
New projection techniques reduce the frequency of projections in solving LCPs.
problem Solving linearly constrained problems efficiently with reduced projection frequency.
method Delayed projection technique to call a projection less frequently.
result Theoretical and practical improvements in convergence rates and efficiency.
Optimization problems with rank constraints arise in many applications, including matrix regression, structured PCA, matrix completion and matrix decomposition problems. An attractive heuristic for solving such problems is to factorize the low-rank matrix, and to run projected gradient descent on the nonconvex factoriz…
Paper introduces P-FGD for online quantile regression models.
problem Training nonparametric additive quantile regression models in online settings.
method Projected functional gradient descent algorithm (P-FGD) for pinball loss.
result P-FGD achieves minimax optimal consistency rate O(t−2s+12s). New algorithm eliminates sign function in PGD attacks, improving performance.
problem Improving robustness of neural networks against adversarial attacks.
method Proposes a new raw gradient descent (RGD) algorithm that eliminates the sign function in PGD attacks.
result The RGD algorithm outperforms PGD and other competitors in various settings.
New method speeds up training of large kernel models.
problem Scaling kernel machines to large datasets and model sizes.
method Delayed projections in Preconditioned Stochastic Gradient Descent (PSGD).
result Significant training speed up over existing methods.
We propose a generic framework based on a new stochastic variance-reduced gradient descent algorithm for accelerating nonconvex low-rank matrix recovery. Starting from an appropriate initial estimator, our proposed algorithm performs projected gradient descent based on a novel semi-stochastic gradient specifically desi…
New analysis shows SNG's effectiveness in small samples.
problem Limited understanding of SNG in small data settings.
method Sketch-and-project analysis of SNG.
result Global convergence and rate characterization for SNG.
We consider stochastic strongly convex optimization with a complex inequality constraint. This complex inequality constraint may lead to computationally expensive projections in algorithmic iterations of the stochastic gradient descent~(SGD) methods. To reduce the computation costs pertaining to the projections, we pro…
In this paper we study the performance of the Projected Gradient Descent(PGD) algorithm for ℓp-constrained least squares problems that arise in the framework of Compressed Sensing. Relying on the Restricted Isometry Property, we provide convergence guarantees for this algorithm for the entire range of $0\leq p\…
Zeroth-order optimization methods lack inherent privacy guarantees.
problem Ensuring differential privacy in zeroth-order optimization methods.
method Analyzing ZO-GD with and without random initialization for convex and strongly convex objectives.
result ZO-GD is not differentially private for strongly convex objectives and can have superlinear privacy loss.
Algorithm selects public datasets for private machine learning.
problem Choosing the most suitable public dataset for private machine learning.
method Measures gradient subspace distance between public and private datasets.
result Excess risk scales with the subspace distance between gradients.
APGD algorithm reconstructs point set from partial distance measurements.
problem Reconstructing point set configuration from partial Euclidean distance measurements.
method Asymmetric Projected Gradient Descent (APGD) for EDMC problem.
result Global convergence and exact recovery with O(μ2r3κ2nlogn) observations. We model how Lipschitz continuity changes during neural network training.
problem Understanding how Lipschitz continuity evolves during training.
method We use a system of stochastic differential equations to capture the dynamics of Lipschitz continuity under SGD.
result We identify three factors driving the evolution of Lipschitz continuity: gradient flow projection, gradient noise, and Hessian projection.
This is a report for reproducibility challenge of NeurlIPS 2019 on the paper Competitive Gradient Descent (Schafer et al., 2019). The paper introduces a novel algorithm for the numerical computation of Nash equilibria of competitive two-player games. It avoids oscillatory and divergent behaviours seen in alternating gr…
Projection pursuit model improves Gaussian process regression for high-dimensional data.
problem Scalability issues with traditional Gaussian process models in high dimensions.
method Additive Gaussian process regression with dimension expansion and gradient descent.
result The proposed method approximates more complex functions and outperforms traditional models.
Paper examines convergence rate of PGD for BP objective in inverse problems.
problem Optimizing ill-posed linear inverse problems using BP vs LS.
method Analysis of PGD convergence rate for BP objective, comparison with proximal gradient method.
result PGD converges faster for BP objective due to inherent properties.
Recent years have seen a flurry of activities in designing provably efficient nonconvex procedures for solving statistical estimation problems. Due to the highly nonconvex nature of the empirical loss, state-of-the-art procedures often require proper regularization (e.g. trimming, regularized cost, projection) in order…
Paper proposes efficient algorithms for designing SLOPE penalty sequences.
problem Designing SLOPE penalty sequences is computationally expensive.
method Developed two efficient algorithms: PGD and CD for Gaussian and general data matrices respectively.
result Demonstrated improved mean squared error performance of SLOPE with designed penalties.
Study analyzes error in neural network solving PDEs, providing convergence and error bounds.
problem Error analysis of neural network solving PDEs.
method Three-layer tanh neural network with projected gradient descent (PGD).
result Comprehensive error analysis including approximation, generalization, and optimization errors.
Algorithm improves online canonical correlation analysis.
problem Online canonical correlation analysis.
method Stochastic Scaled-Gradient Descent (SSGD) for minimizing expectation over Riemannian manifolds.
result Achieved optimal one-time-scale algorithm with explicit rate of local asymptotic convergence.
Several convex formulation methods have been proposed previously for statistical estimation with structured sparsity as the prior. These methods often require a carefully tuned regularization parameter, often a cumbersome or heuristic exercise. Furthermore, the estimate that these methods produce might not belong to th…
The paper analyzes EM for Mixtures of Experts and shows its equivalence to projected Mirror Descent.
problem Training Mixtures of Experts (MoE) models.
method Rigorously analyzes Expectation Maximization (EM) for MoE models using a Mirror Descent perspective.
result Derives new convergence results and identifies conditions for local linear convergence.
New inequalities help optimize first-order algorithms for statistical risk analysis.
problem Optimizing first-order iterative algorithms for statistical risk analysis.
method Introducing basic inequalities that connect implicit and explicit regularization.
result The basic inequalities translate the number of iterations into an effective regularization coefficient.