arXiv research

Gradient descent fails to train two-layer ReLU networks, leading to poor performance.

LQF linearizes deep models for better interpretability.

This paper explains a mechanism called phase collapse that improves image classification accuracy.

New algorithms accelerate solving nonlinear matrix decomposition with ReLU.

The paper reveals how clustering in ReLU networks enhances interpretability.

Improved learning bounds for corrupted data using thresholded gradient descent.

CryptoNAS improves PI accuracy by 3.4% with 2.4x less latency.

We develop algorithms to learn non-linear dynamical systems without mixing assumptions.

We propose a new notion of `non-linearity' of a network layer with respect to an input batch that is based on its proximity to a linear system, which is reflected in the non-negative rank of the activation matrix. We measure this non-linearity by applying non-negative factorization to the activation matrix. Considering…

Theoretical analysis of deep neural networks for time series data.

In adversarial attacks to machine-learning classifiers, small perturbations are added to input that is correctly classified. The perturbations yield adversarial examples, which are virtually indistinguishable from the unperturbed input, and yet are misclassified. In standard neural networks used for deep learning, atta…

We propose a new way of thinking about deep neural networks, in which the linear and non-linear components of the network are naturally derived and justified in terms of principles in probability theory. In particular, the models constructed in our framework assign probabilities to uncertain realizations, leading to Ku…

Study on dynamics of non-linear autoencoders learning principal components.

Paper compresses deep neural networks by eliminating redundant neurons.

Deep ReLU networks approximate functions in Sobolev and Besov spaces efficiently.

Adversarial attacks add perturbations to the input features with the intent of changing the classification produced by a machine learning system. Small perturbations can yield adversarial examples which are misclassified despite being virtually indistinguishable from the unperturbed input. Classifiers trained with stan…

SPLASH units improve deep neural network accuracy and adversarial robustness.

New findings on how neural networks generalize with varying dimensions.

`Biologically inspired' activation functions, such as the logistic sigmoid, have been instrumental in the historical advancement of machine learning. However in the field of deep learning, they have been largely displaced by rectified linear units (ReLU) or similar functions, such as its exponential linear unit (ELU) v…

Neural networks extrapolate poorly in simple tasks but succeed in complex ones.

The goal of a recommendation system is to predict the interest of a user in a given item by exploiting the existing set of ratings as well as certain user/item features. A standard approach to modeling this problem is Inductive Matrix Completion where the predicted rating is modeled as an inner product of the user and …

Utilizing recently introduced concepts from statistics and quantitative risk management, we present a general variant of Batch Normalization (BN) that offers accelerated convergence of Neural Network training compared to conventional BN. In general, we show that mean and standard deviation are not always the most appro…

GOLS finds activation functions affect training robustness, especially ReLU.

A corrective neural network approach improves memorization and learning efficiency.

First proper learning algorithm for Gaussian halfspaces with matching sample and computational complexity.

Wide networks with polynomial activations have proven asymptotic behavior.

New algorithms improve neural network verification by exploiting piecewise linear structure.

Study shows depth improves trainability of neural networks by improving kernel conditioning.

This study connects ReLU neural networks to toric geometry to analyze function realization.

The paper investigates how target normalization and momentum affect dying ReLUs in neural networks.

Paper proposes a new activation function to reduce overfitting and large weight update issues.

Single ReLU neuron's gradient dynamics reveal support vectors as key to generalization.

Single neuron with ADA learns XOR and outperforms other functions.

Large deviation principle for deep neural networks with ReLU activation.

This work uses tropical geometry to understand neural network decision boundaries.

Study shows efficient neural network approach for stochastic bandits.

The dying ReLU refers to the problem when ReLU neurons become inactive and only output 0 for any input. There are many empirical and heuristic explanations of why ReLU neurons die. However, little is known about its theoretical analysis. In this paper, we rigorously prove that a deep ReLU network will eventually die in…

Hard to learn ReLU with Gaussian data, but can approximate efficiently.

The paper analyzes deep ReLU CNNs' approximation properties in 2D space.

Proves existence of optimal shallow neural networks with ReLU activation.

Gradient descent learns ReLU functions with non-zero bias efficiently.

Deep ReLU networks can be simplified to a three-layer model.

This article concerns the expressive power of depth in neural nets with ReLU activations and bounded width. We are particularly interested in the following questions: what is the minimal width $w_{\text{min}}(d)$ so that ReLU nets of width $w_{\text{min}}(d)$ (and arbitrary depth) can approximate any continuous functio…

Modified ReLU networks improve regression estimation rates.

Bayesian ReLU nets fix asymptotic overconfidence with infinite features.

Improved bounds on neural network expressivity.

Despite their prevalence in neural networks we still lack a thorough theoretical characterization of ReLU layers. This paper aims to further our understanding of ReLU layers by studying how the activation function ReLU interacts with the linear component of the layer and what role this interaction plays in the success …

Gradient descent converges to minimum Bayes risk for two-layer ReLU networks in mean field regime.