arXiv research

A new method improves stochastic gradient descent for faster and more efficient estimation.

A new method for SVGD reduces variance in high dimensions.

New method improves MMD estimation without convexity assumptions.

FP uses random projections to train networks without feedback, achieving comparable performance to backpropagation.

This is a report for reproducibility challenge of NeurlIPS 2019 on the paper Competitive Gradient Descent (Schafer et al., 2019). The paper introduces a novel algorithm for the numerical computation of Nash equilibria of competitive two-player games. It avoids oscillatory and divergent behaviours seen in alternating gr…

Paper examines convergence rate of PGD for BP objective in inverse problems.

Blind Descent avoids gradient issues, using a different learning approach.

Gradient descent at edge of stability stabilizes implicitly, following projected gradient descent.

This paper proposes robust matrix variate regression models with rank constraints and vector regularization.

We study the problem of supervised linear dimensionality reduction, taking an information-theoretic viewpoint. The linear projection matrix is designed by maximizing the mutual information between the projected signal and the class label (based on a Shannon entropy measure). By harnessing a recent theoretical result on…

A new method for Bayesian inference tackles high-dimensional problems.

A new model for dynamic covariance recovery in neuroimaging data.

We study alignment in linear neural networks and its relation to gradient descent.

We study the projected gradient descent method on low-rank matrix problems with a strongly convex objective. We use the Burer-Monteiro factorization approach to implicitly enforce low-rankness; such factorization introduces non-convexity in the objective. We focus on constraint sets that include both positive semi-defi…

In this paper, we study the problem of constrained robust (min-max) optimization ina black-box setting, where the desired optimizer cannot access the gradients of the objective function but may query its values. We present a principled optimization framework, integrating a zeroth-order (ZO) gradient estimator with an a…

Recent years have seen increased interest in performance guarantees of gradient descent algorithms for non-convex optimization. A number of works have uncovered that gradient noise plays a critical role in the ability of gradient descent recursions to efficiently escape saddle-points and reach second-order stationary p…

Three new efficient algorithms project vectors onto weighted l1 ball.

Paper studies PSGD for constrained optimization problems and its statistical properties.

In this work we introduce a conditional accelerated lazy stochastic gradient descent algorithm with optimal number of calls to a stochastic first-order oracle and convergence rate $O\left(\frac{1}{\varepsilon^2}\right)$ improving over the projection-free, Online Frank-Wolfe based stochastic gradient descent of Hazan an…

AGD converges in polynomial iterations to optimal matrix factorization.

In this paper, we consider the problem of learning high-dimensional tensor regression problems with low-rank structure. One of the core challenges associated with learning high-dimensional models is computation since the underlying optimization problems are often non-convex. While convex relaxations could lead to polyn…

We propose a projected semi-stochastic gradient descent method with mini-batch for improving both the theoretical complexity and practical performance of the general stochastic gradient descent method (SGD). We are able to prove linear convergence under weak strong convexity assumption. This requires no strong convexit…

The paper interprets diffusion models as gradient descent and proposes a new sampler.

A new algorithm estimates sparse gradients on graphs with improved risk bounds.

ScaledGD improves gradient descent for ill-conditioned low-rank matrix estimation.

In this paper, we aim at providing an introduction to the gradient descent based optimization algorithms for learning deep neural network models. Deep learning models involving multiple nonlinear projection layers are very challenging to train. Nowadays, most of the deep learning model training still relies on the back…

Iterative hard thresholding (IHT) is a projected gradient descent algorithm, known to achieve state of the art performance for a wide range of structured estimation problems, such as sparse inference. In this work, we consider IHT as a solution to the problem of learning sparse discrete distributions. We study the hard…

PGD-trained models have a preferential direction in their gradients, which improves robustness.

Alt-GDA outperforms Sim-GDA in minimax games with near-optimal local convergence.

Faster reconstruction of compressed signals using conditional GAN and NPGD.

This paper proposes an alternating back-propagation algorithm for learning the generator network model. The model is a non-linear generalization of factor analysis. In this model, the mapping from the continuous latent factors to the observed signal is parametrized by a convolutional neural network. The alternating bac…

A parameter-free PGD algorithm for convex optimization.

The curse of dimensionality is a longstanding challenge in Bayesian inference in high dimensions. In this work, we propose a projected Stein variational gradient descent (pSVGD) method to overcome this challenge by exploiting the fundamental property of intrinsic low dimensionality of the data informed subspace stemmin…

Stochastic approximation algorithms show exponential progress bounds.

New method speeds up PGD for CV robustness evaluation.

Decentralized optimization on dynamic manifolds with improved regret bound.

This paper deals with unsupervised clustering with feature selection. The problem is to estimate both labels and a sparse projection matrix of weights. To address this combinatorial non-convex problem maintaining a strict control on the sparsity of the matrix of weights, we propose an alternating minimization of the Fr…

Develops first and second-order pseudo-mirror descent methods for nonnegative function estimation.

SAGD uses Langevin algorithm for efficient gradient descent.

Paper proposes algorithms for solving nonconvex-nonconcave problems with complexity guarantees.

Nesterov's extrapolation improves convergence in nonsmooth optimization.

Accelerated optimization methods improve robustness and privacy in estimation.

New algorithm improves learning efficiency in multi-task contextual bandits.

Privacy-preserving SGD with heavy-tailed noise achieves differential privacy guarantees.

SPGD improves adversarial training efficiency and accuracy.

New projection techniques reduce the frequency of projections in solving LCPs.

Optimization problems with rank constraints arise in many applications, including matrix regression, structured PCA, matrix completion and matrix decomposition problems. An attractive heuristic for solving such problems is to factorize the low-rank matrix, and to run projected gradient descent on the nonconvex factoriz…

SoftAD improves classification accuracy with less fine-tuning and fewer computational costs.