arXiv research

Paper addresses fault-tolerance in distributed machine learning with stochastic gradient descent.

Policy gradient methods are very attractive in reinforcement learning due to their model-free nature and convergence guarantees. These methods, however, suffer from high variance in gradient estimation, resulting in poor sample efficiency. To mitigate this issue, a number of variance-reduction approaches have been prop…

Two new regularization methods improve neural network performance and complexity control.

We study the effectiveness of non-uniform randomized feature selection in decision tree classification. We experimentally evaluate two feature selection methodologies, based on information extracted from the provided dataset: $(i)$ \emph{leverage scores-based} and $(ii)$ \emph{norm-based} feature selection. Experimenta…

U-Clip clips gradients to reduce bias without discarding them.

Unified PAC-Bayesian framework for deep learning generalization.

In this paper, we study the problem of estimating the covariance matrix under differential privacy, where the underlying covariance matrix is assumed to be sparse and of high dimensions. We propose a new method, called DP-Thresholding, to achieve a non-trivial $\ell_2$-norm based error bound, which is significantly bet…

This paper is concerned about sparse, continuous frequency estimation in line spectral estimation, and focused on developing gridless sparse methods which overcome grid mismatches and correspond to limiting scenarios of existing grid-based approaches, e.g., $\ell_1$ optimization and SPICE, with an infinitely dense grid…

GraN detects adversarial and misclassified examples efficiently.

Recent studies incorporate Nesterov's accelerated gradient method for the acceleration of gradient based training. The Nesterov's Accelerated Quasi-Newton (NAQ) method has shown to drastically improve the convergence speed compared to the conventional quasi-Newton method. This paper implements NAQ for non-convex optimi…

Solves selecting the best optimizing system problems.

A toolkit for path-norms enhances neural network generalization bounds.

Advancements in parallel processing have lead to a surge in multilayer perceptrons' (MLP) applications and deep learning in the past decades. Recurrent Neural Networks (RNNs) give additional representational power to feedforward MLPs by providing a way to treat sequential data. However, RNNs are hard to train using con…

PathBoost boosts graph-level predictions using path-based features.

We investigate the capacity, convexity and characterization of a general family of norm-constrained feed-forward networks.

PES method reduces bias in gradient estimation for unrolled graphs.

A novel metric and framework for evaluating gradient norm equality in deep neural networks.

Gradient descent with biased rounding errors converges faster under certain conditions.

Paper introduces new risk norms based on ES with flexible distortion functions.

Adaptive-SGD method optimizes machine learning training with dynamic batch and step sizes.

We investigate 1) the rate at which refined properties of the empirical risk---in particular, gradients---converge to their population counterparts in standard non-convex learning tasks, and 2) the consequences of this convergence for optimization. Our analysis follows the tradition of norm-based capacity control. We p…

Two new feature selection algorithms improve on RFE.

SGBD algorithm improves robustness in Bayesian sampling.

COMRADE is a communication-efficient, Byzantine-resilient second-order optimization algorithm.

Joint sparsity offers powerful structural cues for feature selection, especially for variables that are expected to demonstrate a "grouped" behavior. Such behavior is commonly modeled via group-lasso, multitask lasso, and related methods where feature selection is effected via mixed-norms. Several mixed-norm based spar…

In this paper, we theoretically prove that adding one special neuron per output unit eliminates all suboptimal local minima of any deep neural network, for multi-class classification, binary classification, and regression with an arbitrary loss function, under practical assumptions. At every local minimum of any deep n…

Network anomaly detection is still a vibrant research area. As the fast growth of network bandwidth and the tremendous traffic on the network, there arises an extremely challengeable question: How to efficiently and accurately detect the anomaly on multiple traffic? In multi-task learning, the traffic consisting of flo…

Truncated SGD with heavy-tailed noise eliminates sharp local minima.

The paper offers generalization bounds for Transformers that ignore sequence length.

We propose a mixed integer programming (MIP) model and iterative algorithms based on topological orders to solve optimization problems with acyclic constraints on a directed graph. The proposed MIP model has a significantly lower number of constraints compared to popular MIP models based on cycle elimination constraint…

SWRLDA improves LDA for multi-class classification with edge classes.

A wide class of machine learning algorithms can be reduced to variable elimination on factor graphs. While factor graphs provide a unifying notation for these algorithms, they do not provide a compact way to express repeated structure when compared to plate diagrams for directed graphical models. To exploit efficient t…

New algorithm eliminates sign function in PGD attacks, improving performance.

A new framework explains why early pruning works well.

Deep neural networks work well at approximating complicated functions when provided with data and trained by gradient descent methods. At the same time, there is a vast amount of existing functions that programmatically solve different tasks in a precise manner eliminating the need for training. In many cases, it is po…

In this paper, we study the convergence of generative adversarial networks (GANs) from the perspective of the informativeness of the gradient of the optimal discriminative function. We show that GANs without restriction on the discriminative function space commonly suffer from the problem that the gradient produced by …

A new method directly encodes data into latent space using gradient flow.

Adversarial training, in which a network is trained on adversarial examples, is one of the few defenses against adversarial attacks that withstands strong attacks. Unfortunately, the high cost of generating strong adversarial examples makes standard adversarial training impractical on large-scale problems like ImageNet…

EXODUS improves training of SNNs by stabilizing gradients and reducing complexity.

PoWER-BERT speeds up BERT inference by eliminating redundant word-vectors.

Exploiting sparsity enables hardware systems to run neural networks faster and more energy-efficiently. However, most prior sparsity-centric optimization techniques only accelerate the forward pass of neural networks and usually require an even longer training process with iterative pruning and retraining. We observe t…

Backpropagation-free RL method trains layers using local signals.

Quantum annealer speeds up RBM training for image classification.

Extends importance sampling to nonlinear models using adjoint operators.

Many statistical $M$-estimators are based on convex optimization problems formed by the combination of a data-dependent loss function with a norm-based regularizer. We analyze the convergence rates of projected gradient and composite gradient methods for solving such problems, working within a high-dimensional framewor…

A new method computes gradients without backpropagation.

We propose a computationally efficient wrapper feature selection method - called Autoencoder and Model Based Elimination of features using Relevance and Redundancy scores (AMBER) - that uses a single ranker model along with autoencoders to perform greedy backward elimination of features. The ranker model is used to pri…

This paper analyzes convergence of RMSProp and Adam in non-convex optimization with tight complexity bounds.