arXiv research

Recalls and refines the concept of algebraically rectifiable curves.

Deep neural networks enforce non-crossing quantile regression curves.

In this paper we investigate the performance of different types of rectified activation functions in convolutional neural network: standard rectified linear unit (ReLU), leaky rectified linear unit (Leaky ReLU), parametric rectified linear unit (PReLU) and a new randomized leaky rectified linear units (RReLU). We evalu…

Paper proposes a new activation function to reduce overfitting and large weight update issues.

CRITS improves time series classification with interpretable local explanations.

Dropout training improves neural networks' performance.

In this article, we study rectifying curves in arbitrary dimensional Euclidean space. A curve is said to be a rectifying curve if, in all points of the curve, the orthogonal complement of its normal vector contains a fixed point. We characterize rectifying curves in the $n$-dimensional Euclidean space in different ways…

This paper provides a theoretical justification of the superior classification performance of deep rectifier networks over shallow rectifier networks from the geometrical perspective of piecewise linear (PWL) classifier boundaries. We show that, for a given threshold on the approximation error, the required number of b…

The paper characterizes rectifying curves on smooth surfaces using isometries and Darboux frames.

In this article, we show that, for any compact 3-manifold, there is a $C^{1}$ volume-minimizing one-dimensional foliation. More generally, we show the existence of mass-minimizing rectifiable sections of sphere bundles without isolated "pole points" in the base manifold. This same analysis is used to show that the exam…

Methods from convex optimization are widely used as building blocks for deep learning algorithms. However, the reasons for their empirical success are unclear, since modern convolutional networks (convnets), incorporating rectifier units and max-pooling, are neither smooth nor convex. Standard guarantees therefore do n…

AReLU uses attention-based rectification to improve neural network performance.

A neural network with a single hidden layer can't represent certain multivariable functions.

We consider a neural network architecture with randomized features, a sign-splitter, followed by rectified linear units (ReLU). We prove that our architecture exhibits robustness to the input perturbation: the output feature of the neural network exhibits a Lipschitz continuity in terms of the input perturbation. We fu…

In this paper, we introduce transformations of deep rectifier networks, enabling the conversion of deep rectifier networks into shallow rectifier networks. We subsequently prove that any rectifier net of any depth can be represented by a maximum of a number of functions that can be realized by a shallow network with a …

Algorithm learns two-layer residual units using ReLU activations from samples.

Volume of unit balls defined by quadratic differentials is not proper and has integrable volume.

Proposes a non-crossing deep neural network quantile regression method.

We present QuickNet, a fast and accurate network architecture that is both faster and significantly more accurate than other fast deep architectures like SqueezeNet. Furthermore, it uses less parameters than previous networks, making it more memory efficient. We do this by making two major modifications to the referenc…

We analyze dropout in deep networks with rectified linear units and the quadratic loss. Our results expose surprising differences between the behavior of dropout and more traditional regularizers like weight decay. For example, on some simple data sets dropout training produces negative weights even though the output i…

Neural networks solve Knapsack problems with provable guarantees.

Paper proposes efficient image inversion and editing using rectified stochastic differential equations.

This study uses neural networks to solve interpolation problems with sparse, infinitely wide layers.

Unique minimal surfaces near quadratic cones are identified.

Rectified Linear Units (ReLU) have become the main model for the neural units in current deep learning systems. This choice has been originally suggested as a way to compensate for the so called vanishing gradient problem which can undercut stochastic gradient descent (SGD) learning in networks composed of multiple lay…

Multivariate splines linked to infinitely-wide neural networks with improved numerical performance.

Study approximates nonlinear functionals using deep ReLU networks.

We present a probabilistic variant of the recently introduced maxout unit. The success of deep neural networks utilizing maxout can partly be attributed to favorable performance under dropout, when compared to rectified linear units. It however also depends on the fact that each maxout unit performs a pooling operation…

New framework explains deep neural networks using variational spline theory.

We prove a version of Smirnov type theorem and Charatheodory type theorem for a harmonic homeomorphism of the unit disk onto a Jordan surface with rectifiable boundary. Further we establish the classical isoperimetric inequality and Riesz--Zygmund inequality for Jordan harmonic surfaces without any smoothness assumptio…

Activation functions play a key role in providing remarkable performance in deep neural networks, and the rectified linear unit (ReLU) is one of the most widely used activation functions. Various new activation functions and improvements on ReLU have been proposed, but each carry performance drawbacks. In this paper, w…

PHP connects to ReLU neural networks for scalable Bayesian inference.

In this paper, we compute the covolume of the group of units of the quadratic form f_d^n(x) = x_1^2 + x_2^2 + . . . + x_n^2 - d x_{n+1}^2 with d an odd, positive, square-free integer. Mcleod has determined the hyperbolic Coxeter fundamental domain of the reflection subgroup of the group of units of the quadratic form f…

MVRSM optimizes expensive functions with mixed variables, outperforming state-of-the-art methods.

Neural networks with rectified linear unit activations are essentially multivariate linear splines. As such, one of many ways to measure the "complexity" or "expressivity" of a neural network is to count the number of knots in the spline model. We study the number of knots in fully-connected feedforward neural networks…

The paper investigates how target normalization and momentum affect dying ReLUs in neural networks.

Most deep neural networks use simple, fixed activation functions, such as sigmoids or rectified linear units, regardless of domain or network structure. We introduce differential equation units (DEUs), an improvement to modern neural networks, which enables each neuron to learn a particular nonlinear activation functio…

The study evaluates the performance of ANNs in financial forecasting.

A new method prunes neural network channels based on operation characteristics.

The paper provides theoretical guarantees for neural network-based anomaly detection.

Recently, neural networks in machine learning use rectified linear units (ReLUs) in early processing layers for better performance. Training these structures sometimes results in "dying ReLU units" with near-zero outputs. We first explore this condition via simulation using the CIFAR-10 dataset and variants of two popu…

Introduces $\mathbb{K}$-framings for surfaces, generalizing quadratic forms.

The paper calculates bounds on the local Lipschitz constants of neural network layers.

Deep learning method for semiparametric regression of spatial data.

We study layered neural networks of rectified linear units (ReLU) in a modelling framework for stochastic training processes. The comparison with sigmoidal activation functions is in the center of interest. We compute typical learning curves for shallow networks with K hidden units in matching student teacher scenarios…

ADMM algorithm solves nonlinear matrix decompositions efficiently.

We revisit skip-gram negative sampling (SGNS), one of the most popular neural-network based approaches to learning distributed word representation. We first point out the ambiguity issue undermining the SGNS model, in the sense that the word vectors can be entirely distorted without changing the objective value. To res…

We derive Frenet-type results and invariants of spatial curves immersed in $3$-dimensional generalized Minkowski spaces, i.e., in linear spaces which satisfy all axioms of finite dimensional real Banach spaces except for the symmetry axiom. Further on, we characterize cylindrical helices and rectifying curves in such s…