arXiv research

SMD outperforms SGD in over-parametrized linear models for certain data distributions.

Matrix SMD converges to unique solution minimizing Bregman divergence.

Study reveals dynamics of neural networks with normalization, weight decay, and SGD.

Stochastic mirror descent (SMD) is a fairly new family of algorithms that has recently found a wide range of applications in optimization, machine learning, and control. It can be considered a generalization of the classical stochastic gradient algorithm (SGD), where instead of updating the weight vector along the nega…

Most modern learning problems are highly overparameterized, meaning that there are many more parameters than the number of training data points, and as a result, the training loss may have infinitely many global minima (parameter vectors that perfectly interpolate the training data). Therefore, it is important to under…

VRSMD improves SMD convergence and has implicit regularization.

Stochastic descent methods (of the gradient and mirror varieties) have become increasingly popular in optimization. In fact, it is now widely recognized that the success of deep learning is not only due to the special deep architecture of the models, but also due to the behavior of the stochastic descent methods used, …

Spectral mixture (SM) kernels comprise a powerful class of generalized kernels for Gaussian processes (GPs) to describe complex patterns. This paper introduces model compression and time- and phase (TP) modulated dependency structures to the original (SM) kernel for improved generalization of GPs. Specifically, by adop…

New method for training deep neural networks with regularization, converging to better generalization.

Stochastic mirror descent improves performance on ensemble models.

We consider the problem of learning convex aggregation of models, that is as good as the best convex aggregation, for the binary classification problem. Working in the stream based active learning setting, where the active learner has to make a decision on-the-fly, if it wants to query for the label of the point curren…

SMT trains generative models by estimating mixture scores, outperforming existing methods.

Paper proposes ZO-SMD for MERO, achieving optimal convergence rates.

New method estimates mixture distributions using stochastic mirror descent.

The MAP estimate's log-likelihood sub-optimality is hard to bound in general.

Stochastic gradient methods can converge in expectation under heavy-tailed noise.

This work compares OmniAnomaly with PCA for MTSAD, finding PCA can match or outperform OmniAnomaly.

New metrics fail adversarial tests, with some more robust than others.