arXiv research

Piecewise linear activations create many spurious local minima in neural networks.

Equivariant neural networks use symmetry to interpret complex data.

A new complexity measure for neural networks improves upon classical methods.

Neural networks can represent complex piecewise functions efficiently.

In exchange for large quantities of data and processing power, deep neural networks have yielded models that provide state of the art predication capabilities in many fields. However, a lack of strong guarantees on their behaviour have raised concerns over their use in safety-critical applications. A first step to unde…

We study algebraic varieties of ReLU networks to understand their representable functions.

Two new criteria help understand the advantage of deep neural networks.

Simplifies PLNNs to interpretable models for better explainability.

Neural network models improve survival analysis with reduced computation time.

Upper bounds on fixed points in PWL neural networks with hyperplane analysis.

Paper proposes LANN to measure model complexity of neural networks with curve activation functions.

New proof shows efficient ReLU networks for piecewise linear functions.

We present a new, unifying approach following some recent developments on the complexity of neural networks with piecewise linear activations. We treat neural network layers with piecewise linear activations as tropical polynomials, which generalize polynomials in the so-called $(\max, +)$ or tropical algebra, with pos…

Study shows efficient neural network approach for stochastic bandits.

We study the complexity of functions computable by deep feedforward neural networks with piecewise linear activations in terms of the symmetries and the number of linear regions that they have. Deep networks are able to sequentially map portions of each layer's input-space to the same output. In this way, deep models c…

New algorithms optimize neural networks with ReLU activations using sampling.

ReLU neural networks define piecewise linear functions of their inputs. However, initializing and training a neural network is very different from fitting a linear spline. In this paper, we expand empirically upon previous theoretical work to demonstrate features of trained neural networks. Standard network initializat…

Paper introduces $\ell$-DER for regression tasks using morphological operators and convex-concave procedure.

The study estimates the expressiveness of GCNs with bounds on the number of linear regions.

Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piecewise linear activation function that is learned independently for each neuron using gradient descent. With this adaptive activation function, we are able to improve upon deep neural ne…

This work generalizes bounds on the number of linear regions in CPWL NNs.

To help understand the underlying mechanisms of neural networks (NNs), several groups have, in recent years, studied the number of linear regions $\ell$ of piecewise linear functions generated by deep neural networks (DNN). In particular, they showed that $\ell$ can grow exponentially with the number of network paramet…

SGD trains ReLU networks to implement piecewise linear maps with at most 3 knot points.

Deep neural networks with piecewise-polynomial activations can approximate smooth functions and their derivatives.

The developments of deep neural networks (DNN) in recent years have ushered a brand new era of artificial intelligence. DNNs are proved to be excellent in solving very complex problems, e.g., visual recognition and text understanding, to the extent of competing with or even surpassing people. Despite inspiring and enco…

Unified piecewise-linear Ricci flows improve community detection.

Deep neural nets on 1-D data are convex Lasso models with reflection features.

The approximation power of general feedforward neural networks with piecewise linear activation functions is investigated. First, lower bounds on the size of a network are established in terms of the approximation error and network depth and width. These bounds improve upon state-of-the-art bounds for certain classes o…

It is well-known that the expressivity of a neural network depends on its architecture, with deeper networks expressing more complex functions. In the case of networks that compute piecewise linear functions, such as those with ReLU activation, the number of distinct linear regions is a natural measure of expressivity.…

Enhances deep neural networks for MRI reconstruction by increasing expressivity.

Hi-fi priors enhance BNNs by learning flexible activations.

GraN-GAN normalizes gradients for better GAN performance.

Unified approach to verify NN properties using ReLU's unique polytope structure.

Wide networks with polynomial activations have proven asymptotic behavior.

We investigate the complexity of deep neural networks (DNN) that represent piecewise linear (PWL) functions. In particular, we study the number of linear regions, i.e. pieces, that a PWL function represented by a DNN can attain, both theoretically and empirically. We present (i) tighter upper and lower bounds for the m…

New topological complexity measures for neural networks.

This paper tackles discontinuous neural networks for better approximation of piecewise continuous functions.

The current understanding of deep neural networks can only partially explain how input structure, network parameters and optimization algorithms jointly contribute to achieve the strong generalization power that is typically observed in many real-world applications. In order to improve the comprehension and interpretab…

Paper presents neural network controllers for offset-free setpoint tracking.

A deep neural network (DNN) with piecewise linear activations can partition the input space into numerous small linear regions, where different linear functions are fitted. It is believed that the number of these regions represents the expressivity of the DNN. This paper provides a novel and meticulous perspective to l…

GroupSort neural networks can approximate Lipschitz continuous functions.

Convex dual network improves neural network reconstruction for medical imaging.

We can compare the expressiveness of neural networks that use rectified linear units (ReLUs) by the number of linear regions, which reflect the number of pieces of the piecewise linear functions modeled by such networks. However, enumerating these regions is prohibitive and the known analytical bounds are identical for…

The success of Deep Learning and its potential use in many safety-critical applications has motivated research on formal verification of Neural Network (NN) models. In this context, verification involves proving or disproving that an NN model satisfies certain input-output properties. Despite the reputation of learned …

Modified ReLU networks improve regression estimation rates.

Study approximates nonlinear functionals using deep ReLU networks.

Hard problem of learning simple generative models from i.i.d. samples.

Most existing interpretable methods explain a black-box model in a post-hoc manner, which uses simpler models or data analysis techniques to interpret the predictions after the model is learned. However, they (a) may derive contradictory explanations on the same predictions given different methods and data samples, and…