arXiv research

New insights into how linear classifiers and leaky ReLU networks can overfit without harming generalization.

State-of-the-art machine learning models frequently misclassify inputs that have been perturbed in an adversarial manner. Adversarial perturbations generated for a given input and a specific classifier often seem to be effective on other inputs and even different classifiers. In other words, adversarial perturbations s…

New method counts boundary pieces in ReLU classifiers for better complexity measure.

Neural networks can approximate high-dimensional classifiers with ReLU networks under margin conditions.

We study the implicit bias of gradient descent methods in solving a binary classification problem over a linearly separable dataset. The classifier is described by a nonlinear ReLU model and the objective function adopts the exponential loss function. We first characterize the landscape of the loss function and show th…

Neural networks can separate non-separable data using feature maps.

Despite their prevalence in neural networks we still lack a thorough theoretical characterization of ReLU layers. This paper aims to further our understanding of ReLU layers by studying how the activation function ReLU interacts with the linear component of the layer and what role this interaction plays in the success …

It has been shown that neural network classifiers are not robust. This raises concerns about their usage in safety-critical systems. We propose in this paper a regularization scheme for ReLU networks which provably improves the robustness of the classifier by maximizing the linear regions of the classifier as well as t…

A bias classifier is introduced to resist adversarial attacks.

Constructs classifiers for neural networks with specific data configurations.

New insights into how neural networks classify data.

The paper proves neural networks' consistency and optimal convergence rates for various function classes.

The paper analyzes the Rashomon ratio for infinite classifier families and shows its importance for choosing good classifiers.

It has recently been shown that neural networks but also other classifiers are vulnerable to so called adversarial attacks e.g. in object recognition an almost non-perceivable change of the image changes the decision of the classifier. Relatively fast heuristics have been proposed to produce these adversarial inputs bu…

The study uses a ReLU network to discern geometric structure in data via the Data Information Matrix.

Deep neural networks classify unbounded Gaussian mixture data without dimensionality issues.

Paper finds conditions for benign overfitting in neural networks.

We study the necessary and sufficient complexity of ReLU neural networks---in terms of depth and number of weights---which is required for approximating classifier functions in $L^2$. As a model class, we consider the set $\mathcal{E}^β(\mathbb R^d)$ of possibly discontinuous piecewise $C^β$ functions $f : [-1/2, 1/2]^…

New polynomial-time solutions found for training ReLU networks, mirroring Max-Cut complexity.

Classifiers used in the wild, in particular for safety-critical systems, should not only have good generalization properties but also should know when they don't know, in particular make low confidence predictions far away from the training data. We show that ReLU type neural networks which yield a piecewise linear cla…

Deep neural networks achieve optimal classification rates in high dimensions.

Deep neural networks can classify any dataset with exact solutions.

In this paper, we analyze efficacy of the fast gradient sign method (FGSM) and the Carlini-Wagner's L2 (CW-L2) attack. We prove that, within a certain regime, the untargeted FGSM can fool any convolutional neural nets (CNNs) with ReLU activation; the targeted FGSM can mislead any CNNs with ReLU activation to classify a…

Gradient descent amplifies random features in neural networks to useful ones.

Neural networks approximate and estimate binary classifiers with polynomial input dependence.

In adversarial attacks to machine-learning classifiers, small perturbations are added to input that is correctly classified. The perturbations yield adversarial examples, which are virtually indistinguishable from the unperturbed input, and yet are misclassified. In standard neural networks used for deep learning, atta…

Adversarial attacks add perturbations to the input features with the intent of changing the classification produced by a machine learning system. Small perturbations can yield adversarial examples which are misclassified despite being virtually indistinguishable from the unperturbed input. Classifiers trained with stan…

Gradient-trained shallow networks can generalize well but are vulnerable to small-radius adversarial attacks.

Improved generalization bounds for CNNs using Rademacher complexity.

This study connects ReLU neural networks to toric geometry to analyze function realization.

Previous work has questioned the conditions under which the decision regions of a neural network are connected and further showed the implications of the corresponding theory to the problem of adversarial manipulation of classifiers. It has been proven that for a class of activation functions including leaky ReLU, neur…

The paper investigates how target normalization and momentum affect dying ReLUs in neural networks.

Single ReLU neuron's gradient dynamics reveal support vectors as key to generalization.

Study on deep neural networks using concentration inequalities and optimal stopping.

This paper extends neural collapse to class-imbalanced datasets using an unconstrained ReLU feature model.

Large deviation principle for deep neural networks with ReLU activation.

Study shows efficient neural network approach for stochastic bandits.

The dying ReLU refers to the problem when ReLU neurons become inactive and only output 0 for any input. There are many empirical and heuristic explanations of why ReLU neurons die. However, little is known about its theoretical analysis. In this paper, we rigorously prove that a deep ReLU network will eventually die in…

Fine-tuning neural networks to guarantee performance on specific examples can also introduce incorrect inputs.

Finding minimum distortion of adversarial examples and thus certifying robustness in neural network classifiers for given data points is known to be a challenging problem. Nevertheless, recently it has been shown to be possible to give a non-trivial certified lower bound of minimum adversarial distortion, and some rece…

The paper analyzes deep ReLU CNNs' approximation properties in 2D space.

The angular power spectrum characterizes neural network complexity.

Proves existence of optimal shallow neural networks with ReLU activation.

LQF linearizes deep models for better interpretability.

Gradient descent learns ReLU functions with non-zero bias efficiently.

Deep ReLU networks can be simplified to a three-layer model.

This article concerns the expressive power of depth in neural nets with ReLU activations and bounded width. We are particularly interested in the following questions: what is the minimal width $w_{\text{min}}(d)$ so that ReLU nets of width $w_{\text{min}}(d)$ (and arbitrary depth) can approximate any continuous functio…

Modified ReLU networks improve regression estimation rates.