arXiv research

Piecewise linear activations create many spurious local minima in neural networks.

We present a new, unifying approach following some recent developments on the complexity of neural networks with piecewise linear activations. We treat neural network layers with piecewise linear activations as tropical polynomials, which generalize polynomials in the so-called $(\max, +)$ or tropical algebra, with pos…

Hi-fi priors enhance BNNs by learning flexible activations.

Paper proposes LANN to measure model complexity of neural networks with curve activation functions.

Upper bounds on fixed points in PWL neural networks with hyperplane analysis.

Equivariant neural networks use symmetry to interpret complex data.

Simplifies PLNNs to interpretable models for better explainability.

We study algebraic varieties of ReLU networks to understand their representable functions.

This work generalizes bounds on the number of linear regions in CPWL NNs.

Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piecewise linear activation function that is learned independently for each neuron using gradient descent. With this adaptive activation function, we are able to improve upon deep neural ne…

New framework interprets deep neural networks through input structure.

New algorithms optimize neural networks with ReLU activations using sampling.

Wide networks with polynomial activations have proven asymptotic behavior.

A new complexity measure for neural networks improves upon classical methods.

We study the complexity of functions computable by deep feedforward neural networks with piecewise linear activations in terms of the symmetries and the number of linear regions that they have. Deep networks are able to sequentially map portions of each layer's input-space to the same output. In this way, deep models c…

The approximation power of general feedforward neural networks with piecewise linear activation functions is investigated. First, lower bounds on the size of a network are established in terms of the approximation error and network depth and width. These bounds improve upon state-of-the-art bounds for certain classes o…

Deep neural nets on 1-D data are convex Lasso models with reflection features.

This paper shows that every sublevel set of the loss function of a class of deep over-parameterized neural nets with piecewise linear activation functions is connected and unbounded. This implies that the loss has no bad local valleys and all of its global minima are connected within a unique and potentially very large…

The developments of deep neural networks (DNN) in recent years have ushered a brand new era of artificial intelligence. DNNs are proved to be excellent in solving very complex problems, e.g., visual recognition and text understanding, to the extent of competing with or even surpassing people. Despite inspiring and enco…

Efficiently checks local robustness in neural networks using geometric projections.

The study estimates the expressiveness of GCNs with bounds on the number of linear regions.

GraN-GAN normalizes gradients for better GAN performance.

We introduce a variational framework to learn the activation functions of deep neural networks. Our aim is to increase the capacity of the network while controlling an upper-bound of the actual Lipschitz constant of the input-output relation. To that end, we first establish a global bound for the Lipschitz constant of …

Two new criteria help understand the advantage of deep neural networks.

New MIP approach for efficient change-point detection.

The center of a quotient group of piecewise linear homeomorphisms is trivial.

Study shows efficient neural network approach for stochastic bandits.

This article provides an attempt to extend concepts from the theory of Riemannian manifolds to piecewise linear spaces. In particular we propose an analogue of the Ricci tensor, which we give the name of an Einstein vector field. On a given set of piecewise linear spaces we define and discuss (normalized) Ricci flows. …

Global approximation for piecewise linear paths via signatures.

We prove that every piecewise linear manifold of dimension up to four on which a finite group acts by piecewise linear homeomorphisms admits a compatible smooth structure with respect to which the group acts smoothly. This solves a challenge posed by Thurston in dimension three and confirms a conjecture by Kwasik and L…

Max-affine regression method converges linearly using GD and SGD.

Signature uniquely identifies piecewise linear surfaces up to thin homotopy.

Unified approach to verify NN properties using ReLU's unique polytope structure.

PARC uses piecewise linear predictors for regression and classification.

In this paper, we introduce a bordism category $\mathcal{C}_d^{PL}$ whose objects are bundles of closed $(d-1)$-dimensional piecewise linear manifolds and whose morphisms are bundles of $d$-dimensional piecewise linear cobordisms. In the main theorem of this article, we show that the classifying space $B\mathcal{C}_d^{…

We propose to optimize the activation functions of a deep neural network by adding a corresponding functional regularization to the cost function. We justify the use of a second-order total-variation criterion. This allows us to derive a general representer theorem for deep neural networks that makes a direct connectio…

Study approximates nonlinear functionals using deep ReLU networks.

Enhances deep neural networks for MRI reconstruction by increasing expressivity.

DAMI uses interpretable regions to select informative samples for deep learning models.

Paper presents ABGD for efficient piecewise linear regression in high dimensions.

We consider smooth isotropic immersions from the 2-dimensional torus into $R^{2n}$, for $n \geq 2$. When $n = 2$ the image of such map is an immersed Lagrangian torus of $R^4$. We prove that such isotropic immersions can be approximated by arbitrarily $C^0$-close piecewise linear isotropic maps. If $n \geq 3$ the piece…

New cones in 4D space found with minimal mass.

New proof shows efficient ReLU networks for piecewise linear functions.

Paper proposes variational inference for piecewise-linear systems.

The paper tackles prescribing discrete Gaussian curvature on polyhedral surfaces.

First explicit isometric immersion of a flat Klein bottle in 3D space.

It is well-known that the expressivity of a neural network depends on its architecture, with deeper networks expressing more complex functions. In the case of networks that compute piecewise linear functions, such as those with ReLU activation, the number of distinct linear regions is a natural measure of expressivity.…

Hilbert initiated the standpoint in foundations of mathematics. From this standpoint, we allow only a finite number of repetitions of elementary operations when we construct objects and morphisms. When we start from a subset of a Euclidean space. Then we assume that any element of the line has only a finite number of c…