arXiv research

A new method uses Coordinate Descent to optimize ResNet networks for private inference.

BCD algorithm finds global minima in neural networks.

Novel approach analyzes ReLU networks' training dynamics and proposes GmP for improved optimization.

By lifting the ReLU function into a higher dimensional space, we develop a smooth multi-convex formulation for training feed-forward deep neural networks (DNNs). This allows us to develop a block coordinate descent (BCD) training algorithm consisting of a sequence of numerically well-behaved convex optimizations. Using…

High-dimensional models trained on smooth manifolds achieve optimal rates in Wasserstein metrics.

A novel algorithm converges for solving a specific matrix decomposition problem.

Derives equations for deep learning biases and weights, showing data complexity reduction.

Neural networks struggle with learning fixed parities.

This study connects ReLU neural networks to toric geometry to analyze function realization.

The paper investigates how target normalization and momentum affect dying ReLUs in neural networks.

Single ReLU neuron's gradient dynamics reveal support vectors as key to generalization.

Large deviation principle for deep neural networks with ReLU activation.

We seek to improve the data efficiency of neural networks and present novel implementations of parameterized piece-wise polynomial activation functions. The parameters are the y-coordinates of n+1 Chebyshev nodes per hidden unit and Lagrangian interpolation between the nodes produces the polynomial on [-1, 1]. We show …

Study shows efficient neural network approach for stochastic bandits.

The dying ReLU refers to the problem when ReLU neurons become inactive and only output 0 for any input. There are many empirical and heuristic explanations of why ReLU neurons die. However, little is known about its theoretical analysis. In this paper, we rigorously prove that a deep ReLU network will eventually die in…

The paper analyzes deep ReLU CNNs' approximation properties in 2D space.

Proves existence of optimal shallow neural networks with ReLU activation.

Gradient descent learns ReLU functions with non-zero bias efficiently.

Deep ReLU networks can be simplified to a three-layer model.

This article concerns the expressive power of depth in neural nets with ReLU activations and bounded width. We are particularly interested in the following questions: what is the minimal width $w_{\text{min}}(d)$ so that ReLU nets of width $w_{\text{min}}(d)$ (and arbitrary depth) can approximate any continuous functio…

Modified ReLU networks improve regression estimation rates.

Bayesian ReLU nets fix asymptotic overconfidence with infinite features.

We consider the problem of computing the best-fitting ReLU with respect to square-loss on a training set when the examples have been drawn according to a spherical Gaussian distribution (the labels can be arbitrary). Let $\mathsf{opt} < 1$ be the population loss of the best-fitting ReLU. We prove: 1. Finding a ReLU wit…

Improved bounds on neural network expressivity.

Despite their prevalence in neural networks we still lack a thorough theoretical characterization of ReLU layers. This paper aims to further our understanding of ReLU layers by studying how the activation function ReLU interacts with the linear component of the layer and what role this interaction plays in the success …

Gradient descent converges to minimum Bayes risk for two-layer ReLU networks in mean field regime.

Transformers use ReLUs to approximate softmax efficiently.

PHP connects to ReLU neural networks for scalable Bayesian inference.

We study the approximation properties of random ReLU features through their reproducing kernel Hilbert space (RKHS). We first prove a universality theorem for the RKHS induced by random features whose feature maps are of the form of nodes in neural networks. The universality result implies that the random ReLU features…

Paper analyzes GLM-tron for high-dimensional ReLU regression, providing upper and lower bounds.

The paper explores how ReLU DNNs can represent MPC policies and vice versa.

We study algebraic varieties of ReLU networks to understand their representable functions.

Gradient descent with logistic loss can interpolate deep networks with smoothed ReLU activations under certain conditions.

The paper calculates upper bounds on ReLU network Lipschitz constants.

Study approximates nonlinear functionals using deep ReLU networks.

New maximum score estimators using ReLU functions and deep neural networks.

Neural networks with REctified Linear Unit (ReLU) activation functions (a.k.a. ReLU networks) have achieved great empirical success in various domains. Nonetheless, existing results for learning ReLU networks either pose assumptions on the underlying data distribution being e.g. Gaussian, or require the network size an…

Gradient descent biases towards stable rank networks for nearly-orthogonal data.

The paper reveals how clustering in ReLU networks enhances interpretability.

Leaky ReLU activations improve the calibration of Bayesian neural networks.

Polynomial time algorithm learns depth-2 neural networks with ReLU activations.

Algorithm learns arbitrary ReLU neurons under Gaussian inputs.

The paper examines topological features of ReLU networks and their relation to decision boundaries and training loss.

Unified approach to verify NN properties using ReLU's unique polytope structure.

Adam achieves optimal convergence in deep ReLU networks via novel Kakeya bounds.

This paper presents a basic property of region dividing of ReLU (rectified linear unit) deep learning when new layers are successively added, by which two new perspectives of interpreting deep learning are given. The first is related to decision trees and forests; we construct a deep learning structure equivalent to a …

Deep ReLU networks need Ω(N) parameters to interpolate at irregularly spaced points.

In this paper, we prove that a shallow neural network with a monotone sigmoid, ReLU, ELU, Softplus, or LeakyReLU activation function can arbitrarily well approximate any L^p(p>=2) integrable functions defined on R*[0,1]^n. We also prove that a shallow neural network with a sigmoid, ReLU, ELU, Softplus, or LeakyReLU act…