arXiv research

The paper studies multiple descent in multi-component prediction models.

MT-SGD samples from multiple target distributions using gradient descent.

Paper proves multiplicative weight updates can train neural networks without learning rate tuning.

Unified analysis of generalization curves in large models using gradient flow.

New method uses multiple kernels to improve SVGD performance.

Sparse GCA finds linear relationships in multiple datasets, using gradient descent.

SimpleMKKM improves multi-kernel clustering efficiency.

New method finds linear relationships across multiple data blocks using proximal gradient descent with $\ell_1$ constraint.

Accelerates coordinate descent methods for machine learning problems.

MWGraD solves multi-objective distributional optimization using particle-based gradient descent.

The paper extends kernel ridge regression to product kernels and reveals new convergence behaviors.

In this paper, we aim at providing an introduction to the gradient descent based optimization algorithms for learning deep neural network models. Deep learning models involving multiple nonlinear projection layers are very challenging to train. Nowadays, most of the deep learning model training still relies on the back…

We develop a progressive training approach for neural networks which adaptively grows the network structure by splitting existing neurons to multiple off-springs. By leveraging a functional steepest descent idea, we derive a simple criterion for deciding the best subset of neurons to split and a splitting gradient for …

The Column Subset Selection Problem (CSSP) and the Nyström method are among the leading tools for constructing small low-rank approximations of large datasets in machine learning and scientific computing. A fundamental question in this area is: how well can a data subset of size k compete with the best rank k approxima…

Recent literature has demonstrated promising results for training Generative Adversarial Networks by employing a set of discriminators, in contrast to the traditional game involving one generator against a single adversary. Such methods perform single-objective optimization on some simple consolidation of the losses, e…

SGD and stochastic gradient descent converge at optimal rates for certain non-convex functions.

The (stochastic) gradient descent and the multiplicative update method are probably the most popular algorithms in machine learning. We introduce and study a new regularization which provides a unification of the additive and multiplicative updates. This regularization is derived from an hyperbolic analogue of the entr…

The purpose of this note is introduce a new axiom (called the Descent Axiom) in the theory of $r$-spin cohomological field theories. This axiom explains the origin of gravitational descendants in this theory. Furthermore, the Descent Axiom immediately implies the Vanishing Axiom, explicating the latter (which has no a …

Two novel algorithms improve distributed machine learning in the presence of Byzantine adversaries.

Study shows double and triple descent in unsupervised autoencoders, improving performance in various tasks.

Inspired by dynamic programming, we propose Stochastic Virtual Gradient Descent (SVGD) algorithm where the Virtual Gradient is defined by computational graph and automatic differentiation. The method is computationally efficient and has little memory requirements. We also analyze the theoretical convergence properties …

New bounds for M-SGD show its error distribution is nearly Gaussian.

Least squares regression shows unexpected double descent in under-parameterized models.

The paper shows how the generalization curve can have multiple peaks, influenced by data and learning algorithm biases.

Multi-label learning studies the problem where an instance is associated with a set of labels. By treating single-label learning problem as one task, the multi-label learning problem can be casted as solving multiple related tasks simultaneously. In this paper, we propose a novel Multi-task Gradient Descent (MGD) algor…

We provide theoretical complexity analysis for new algorithms to compute the optimal transport (OT) distance between two discrete probability distributions, and demonstrate their favorable practical performance over state-of-art primal-dual algorithms and their capability in solving other problems in large-scale, such …

Differentially private random block coordinate descent improves utility in machine learning.

Natural gradient learning improves synaptic plasticity in spiking neurons.

Gradient descent biases neural networks to use an average of features, leading to non-robustness.

Study on RF regression with SGD shows double descent phenomenon.

Deep learning models can generalize well even when they fit training data perfectly.

New insights into training machine learning models with momentum.

We study the problem of learning one-hidden-layer neural networks with Rectified Linear Unit (ReLU) activation function, where the inputs are sampled from standard Gaussian distribution and the outputs are generated from a noisy teacher network. We analyze the performance of gradient descent for training such kind of n…

Gradient descent and its many variants, including mini-batch stochastic gradient descent, form the algorithmic foundation of modern large-scale machine learning. Due to the size and scale of modern data, gradient computations are often distributed across multiple compute nodes. Unfortunately, such distributed implement…

Despite the wide use of machine learning in adversarial settings including computer security, recent studies have demonstrated vulnerabilities to evasion attacks---carefully crafted adversarial samples that closely resemble legitimate instances, but cause misclassification. In this paper, we examine the adequacy of the…

The paper studies the minimum ℓ₁-norm interpolator's risk behavior in over-parameterized settings.

New framework improves variational inference with Markov chain methods.

In this work, we consider the use of model-driven deep learning techniques for massive multiple-input multiple-output (MIMO) detection. Compared with conventional MIMO systems, massive MIMO promises improved spectral efficiency, coverage and range. Unfortunately, these benefits are coming at the cost of significantly i…

Transformers learn to recall with non-orthogonal embeddings in realistic settings.

Paper proposes a pre-conditioning technique to speed up gradient-descent convergence in distributed linear least-squares problems.

This paper considers the problem of solving systems of quadratic equations, namely, recovering an object of interest $\mathbf{x}^{\natural}\in\mathbb{R}^{n}$ from $m$ quadratic equations/samples $y_{i}=(\mathbf{a}_{i}^{\top}\mathbf{x}^{\natural})^{2}$, $1\leq i\leq m$. This problem, also dubbed as phase retrieval, span…

We introduce a collaborative learning framework allowing multiple parties having different sets of attributes about the same user to jointly build models without exposing their raw data or model parameters. In particular, we propose a Federated Stochastic Block Coordinate Descent (FedBCD) algorithm, in which each party…

Gradient Descent (GD) approximators often fail in the solution space with multiple scales of convexities, i.e., in subspace learning and neural network scenarios. To handle that, one solution is to run GD multiple times from different randomized initial states and select the best solution over all experiments. However,…

To improve the off-sample generalization of classical procedures minimizing the empirical risk under potentially heavy-tailed data, new robust learning algorithms have been proposed in recent years, with generalized median-of-means strategies being particularly salient. These procedures enjoy performance guarantees in …

Stein variational gradient descent (SVGD) was recently proposed as a general purpose nonparametric variational inference algorithm [Liu & Wang, NIPS 2016]: it minimizes the Kullback-Leibler divergence between the target distribution and its approximation by implementing a form of functional gradient descent on a reprod…

A new decomposition explains over-parameterized models' counterintuitive behaviors.

New algorithm reduces complexity for SPD manifold optimization.

We study the risk of minimum-norm interpolants of data in Reproducing Kernel Hilbert Spaces. Our upper bounds on the risk are of a multiple-descent shape for the various scalings of $d = n^α$, $α\in(0,1)$, for the input dimension $d$ and sample size $n$. Empirical evidence supports our finding that minimum-norm interpo…