arXiv research

New algorithm eliminates sign function in PGD attacks, improving performance.

Can health entities collaboratively train deep learning models without sharing sensitive raw data? This paper proposes several configurations of a distributed deep learning method called SplitNN to facilitate such collaborations. SplitNN does not share raw data or model details with collaborating institutions. The prop…

Federated learning (FL) provides a communication-efficient approach to solve machine learning problems concerning distributed data, without sending raw data to a central server. However, existing works on FL only utilize first-order gradient descent (GD) and do not consider the preceding iterations to gradient update w…

A framework for collaborative learning reduces communication rounds.

Study mini-batch SGD noise and its limits, proving complexity guarantees.

We survey distributed deep learning models for training or inference without accessing raw data from clients. These methods aim to protect confidential patterns in data while still allowing servers to train models. The distributed deep learning methods of federated learning, split learning and large batch stochastic gr…

In this work we propose a method for learning wavelet filters directly from data. We accomplish this by framing the discrete wavelet transform as a modified convolutional neural network. We introduce an autoencoder wavelet transform network that is trained using gradient descent. We show that the model is capable of le…

Study finds non-IID data causes FL performance issues.

Federated CTMC model estimates bridge deterioration hazards without sharing raw data.

Federated learning improves by unbiased gradient aggregation and controllable meta updating.

Emerging technologies and applications including Internet of Things (IoT), social networking, and crowd-sourcing generate large amounts of data at the network edge. Machine learning models are often built from the collected data, to enable the detection, classification, and prediction of future events. Due to bandwidth…

Blind Descent avoids gradient issues, using a different learning approach.

Develops a simulation-based method to translate expert knowledge into prior distributions for Bayesian models.

FedDAG learns DAG structures from decentralized data.

In this note, we observe the behavior of gradient flow and discrete and noisy gradient descent in some simple settings. It is commonly noted that addition of noise to gradient descent can affect the trajectory of gradient descent. Here, we run some computer experiments for gradient descent on some simple functions, and…

Stochastic gradient descent on manifolds improves low-rank approximation.

A new method improves stochastic gradient descent for faster and more efficient estimation.

Gradient descent implicitly regularizes neural networks by penalizing large loss gradients.

Derives Mirror Descent from gradient flow on a Riemannian manifold.

We consider the behavior of gradient flow and of discrete and noisy gradient descent. It is commonly noted that the addition of noise to the process of discrete gradient descent can affect the trajectory of gradient descent. In previous work, we observed such effects. There, we considered the case where the minima had …

Gradient descent variants improve phase retrieval accuracy.

Reparameterizes mirror descent as gradient descent for efficient sparse learning.

Gradient descent dynamics in nonconvex models explained with universality.

Gravilon improves gradient descent for neural networks.

Secure social recommendation framework using secret sharing.

SGD reduces test error by decorrelating updates.

New adaptive step-size method for convex optimization without tuning.

Natural gradient descent avoids the magic of model parametrization, leading to different optimization outcomes.

Gradient descent biases towards stable rank networks for nearly-orthogonal data.

Any gradient descent optimization requires to choose a learning rate. With deeper and deeper models, tuning that learning rate can easily become tedious and does not necessarily lead to an ideal convergence. We propose a variation of the gradient descent algorithm in the which the learning rate is not fixed. Instead, w…

Gradient descent can take exponentially long to escape saddle points in 2D.

SGD and stochastic gradient descent converge at optimal rates for certain non-convex functions.

Gradient descent at edge of stability stabilizes implicitly, following projected gradient descent.

This work proposes a new method for variational inference using Wasserstein gradient descent.

Gradient descent achieves exact linear convergence rate for symmetric matrix completion.

New bounds for model generalization under deterministic gradient descent.

We study the problem of training deep neural networks with Rectified Linear Unit (ReLU) activation function using gradient descent and stochastic gradient descent. In particular, we study the binary classification problem and show that for a broad family of loss functions, with proper random weight initialization, both…

Gradient descent with logistic loss can make two-layer networks interpolate binary classification data.

Gradient descent benefits from tangent kernel advantages under specific conditions.

Accelerates coordinate descent methods for machine learning problems.

Gradient descent with large momentum finds flatter minima.

Proposes a continuous flow model to understand and control instability in gradient descent for deep learning.

Gradient descent on MMD GAN parameter space converges globally to target distribution.

Gradient descent methods for deep ReLU networks achieve optimal generalization rates.

Gradient descent on normalized networks reveals sparsity preferences.

Gradient descent fails to learn simple neural networks efficiently.

Proof given for SGD convergence in a concise manner.

In this paper, we aim at providing an introduction to the gradient descent based optimization algorithms for learning deep neural network models. Deep learning models involving multiple nonlinear projection layers are very challenging to train. Nowadays, most of the deep learning model training still relies on the back…