arXiv research

Piecewise linear activations create many spurious local minima in neural networks.

We study algebraic varieties of ReLU networks to understand their representable functions.

We present a new, unifying approach following some recent developments on the complexity of neural networks with piecewise linear activations. We treat neural network layers with piecewise linear activations as tropical polynomials, which generalize polynomials in the so-called $(\max, +)$ or tropical algebra, with pos…

In exchange for large quantities of data and processing power, deep neural networks have yielded models that provide state of the art predication capabilities in many fields. However, a lack of strong guarantees on their behaviour have raised concerns over their use in safety-critical applications. A first step to unde…

Paper proposes LANN to measure model complexity of neural networks with curve activation functions.

Upper bounds on fixed points in PWL neural networks with hyperplane analysis.

Hi-fi priors enhance BNNs by learning flexible activations.

Equivariant neural networks use symmetry to interpret complex data.

This work generalizes bounds on the number of linear regions in CPWL NNs.

We develop a method to learn neural network activations with controlled Lipschitz constant.

Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piecewise linear activation function that is learned independently for each neuron using gradient descent. With this adaptive activation function, we are able to improve upon deep neural ne…

Simplifies PLNNs to interpretable models for better explainability.

GraN-GAN normalizes gradients for better GAN performance.

A new complexity measure for neural networks improves upon classical methods.

New framework interprets deep neural networks through input structure.

New algorithms optimize neural networks with ReLU activations using sampling.

Wide networks with polynomial activations have proven asymptotic behavior.

The study estimates the expressiveness of GCNs with bounds on the number of linear regions.

We study the complexity of functions computable by deep feedforward neural networks with piecewise linear activations in terms of the symmetries and the number of linear regions that they have. Deep networks are able to sequentially map portions of each layer's input-space to the same output. In this way, deep models c…

Deep neural networks with piecewise-polynomial activations can approximate smooth functions and their derivatives.

The developments of deep neural networks (DNN) in recent years have ushered a brand new era of artificial intelligence. DNNs are proved to be excellent in solving very complex problems, e.g., visual recognition and text understanding, to the extent of competing with or even surpassing people. Despite inspiring and enco…

Algorithm identifies best arm in piecewise stationary linear bandits with minimal samples.

Efficiently checks local robustness in neural networks using geometric projections.

The approximation power of general feedforward neural networks with piecewise linear activation functions is investigated. First, lower bounds on the size of a network are established in terms of the approximation error and network depth and width. These bounds improve upon state-of-the-art bounds for certain classes o…

Deep neural nets on 1-D data are convex Lasso models with reflection features.

Two new criteria help understand the advantage of deep neural networks.

This paper shows that every sublevel set of the loss function of a class of deep over-parameterized neural nets with piecewise linear activation functions is connected and unbounded. This implies that the loss has no bad local valleys and all of its global minima are connected within a unique and potentially very large…

Develops active learning for Jump Gaussian Process models.

We propose a new active learning algorithm for parametric linear regression with random design. We provide finite sample convergence guarantees for general distributions in the misspecified model. This is the first active learner for this setting that provably can improve over passive learning. Unlike other learning se…

The importance of interpretability of machine learning models has been increasing due to emerging enterprise predictive analytics, threat of data privacy, accountability of artificial intelligence in society, and so on. Piecewise linear models have been actively studied to achieve both accuracy and interpretability. Th…

Study shows efficient neural network approach for stochastic bandits.

The center of a quotient group of piecewise linear homeomorphisms is trivial.

New MIP approach for efficient change-point detection.

This article provides an attempt to extend concepts from the theory of Riemannian manifolds to piecewise linear spaces. In particular we propose an analogue of the Ricci tensor, which we give the name of an Einstein vector field. On a given set of piecewise linear spaces we define and discuss (normalized) Ricci flows. …

Global approximation for piecewise linear paths via signatures.

PARC uses piecewise linear predictors for regression and classification.

Investigates stability of piecewise flat Ricci flow using analysis and simulations.

Study approximates nonlinear functionals using deep ReLU networks.

We prove that every piecewise linear manifold of dimension up to four on which a finite group acts by piecewise linear homeomorphisms admits a compatible smooth structure with respect to which the group acts smoothly. This solves a challenge posed by Thurston in dimension three and confirms a conjecture by Kwasik and L…

We can compare the expressiveness of neural networks that use rectified linear units (ReLUs) by the number of linear regions, which reflect the number of pieces of the piecewise linear functions modeled by such networks. However, enumerating these regions is prohibitive and the known analytical bounds are identical for…

Signature uniquely identifies piecewise linear surfaces up to thin homotopy.

Max-affine regression method converges linearly using GD and SGD.

The paper extends a variance gamma model to quadratic functions, reducing arbitrage and computational costs.

Nonlinearity is crucial to the performance of a deep (neural) network (DN). To date there has been little progress understanding the menagerie of available nonlinearities, but recently progress has been made on understanding the rôle played by piecewise affine and convex nonlinearities like the ReLU and absolute value …

It is well-known that the expressivity of a neural network depends on its architecture, with deeper networks expressing more complex functions. In the case of networks that compute piecewise linear functions, such as those with ReLU activation, the number of distinct linear regions is a natural measure of expressivity.…

In this paper, we introduce a bordism category $\mathcal{C}_d^{PL}$ whose objects are bundles of closed $(d-1)$-dimensional piecewise linear manifolds and whose morphisms are bundles of $d$-dimensional piecewise linear cobordisms. In the main theorem of this article, we show that the classifying space $B\mathcal{C}_d^{…

First explicit isometric immersion of a flat Klein bottle in 3D space.

We propose to optimize the activation functions of a deep neural network by adding a corresponding functional regularization to the cost function. We justify the use of a second-order total-variation criterion. This allows us to derive a general representer theorem for deep neural networks that makes a direct connectio…