arXiv research

We propose K-TanH, a novel, highly accurate, hardware efficient approximation of popular activation function TanH for Deep Learning. K-TanH consists of parameterized low-precision integer operations, such as, shift and add/subtract (no floating point operation needed) where parameters are stored in very small look-up t…

Finite-precision learning of $anh$ networks is limited by the Monte Carlo rate.

Many neural networks use the tanh activation function, however when given a probability distribution as input, the problem of computing the output distribution in neural networks with tanh activation has not yet been addressed. One important example is the initialization of the echo state network in reservoir computing…

In this paper we show all possible ramps where an object can move with constant speed under the effect of gravity and friction. The planar ramp are very easy to describe, just rotate a curve with velocity vector (tanh(as),sech(as)). Recall that tanh(as)^2+sech^2(as) = 1. Therefore, the solution of the planar constant s…

The Rectified Linear Unit (ReLU) is a foundational activation function in artficial neural networks. Recent literature frequently misattributes its origin to the 2018 (initial) version of this paper, which exclusively investigated ReLU at the classification layer. This paper formally corrects the citation record by tra…

Bayesian neural networks show good correlation between out-of-sample performance and Bayesian evidence.

New theory explains signal propagation in normalization-free transformers.

Random feature maps improve forecasting of chaotic dynamical systems.

New findings on neural network identifiability using affine symmetries.

Investigates least squares estimation in deterministic MoE models.

Finding minimum distortion of adversarial examples and thus certifying robustness in neural network classifiers for given data points is known to be a challenging problem. Nevertheless, recently it has been shown to be possible to give a non-trivial certified lower bound of minimum adversarial distortion, and some rece…

The paper analyzes the score field of diffusion models using Burgers dynamics.

New method improves training of PINNs for PDEs by adding noisy supervision terms.

New PINNs method improves accuracy in computing Mean Escape Time from bounded domains.

We present a probabilistic framework for nonlinearities, based on doubly truncated Gaussian distributions. By setting the truncation points appropriately, we are able to generate various types of nonlinearities within a unified framework, including sigmoid, tanh and ReLU, the most commonly used nonlinearities in neural…

Upper bounds on fixed points in PWL neural networks with hyperplane analysis.

PDEs constrain smooth functions in neural networks.

We present a Statistical Mechanics (SM) model of deep neural networks, connecting the energy-based and the feed forward networks (FFN) approach. We infer that FFN can be understood as performing three basic steps: encoding, representation validation and propagation. From the meanfield solution of the model, we obtain a…

New insights into neural network training efficiency.

We seek to improve the data efficiency of neural networks and present novel implementations of parameterized piece-wise polynomial activation functions. The parameters are the y-coordinates of n+1 Chebyshev nodes per hidden unit and Lagrangian interpolation between the nodes produces the polynomial on [-1, 1]. We show …

The exclusive or (xor) function is one of the simplest examples that illustrate why nonlinear feedforward networks are superior to linear regression for machine learning applications. We review the xor representation and approximation problems and discuss their solutions in terms of probabilistic logic and associative …

SGD converges globally to logistic loss minima for two-layer nets.

Randomly initialized wide neural networks with zero-mean activations are nearly independent, potentially solving AI interpretability limits.

Study fractal and regular geometry in deep neural networks.

Let H denote the standard one-point completion of a real Hilbert space. Given any non-trivial proper sub-set U of H one may define the so-called `Apollonian' metric d_U on U. When U \subset V \subset H are nested proper subsets we show that their associated Apollonian metrics satisfy the following uniform contraction p…

Previous work has questioned the conditions under which the decision regions of a neural network are connected and further showed the implications of the corresponding theory to the problem of adversarial manipulation of classifiers. It has been proven that for a class of activation functions including leaky ReLU, neur…

Neural networks struggle with periodic functions, a new activation fixes this.

Deep Learning is applied to energy markets to predict extreme loads observed in energy grids. Forecasting energy loads and prices is challenging due to sharp peaks and troughs that arise due to supply and demand fluctuations from intraday system constraints. We propose deep spatio-temporal models and extreme value theo…

This paper proposes a novel fault diagnosis approach based on generative adversarial networks (GAN) for imbalanced industrial time series where normal samples are much larger than failure cases. We combine a well-designed feature extractor with GAN to help train the whole network. Aimed at obtaining data distribution a…

We investigate the loss surface of neural networks. We prove that even for one-hidden-layer networks with "slightest" nonlinearity, the empirical risks have spurious local minima in most cases. Our results thus indicate that in general "no spurious local minima" is a property limited to deep linear networks, and insigh…

A celebrated conjecture due to De Giorgi states that any bounded solution of the equation $Δu + (1-u^2) u = 0 \hbox{in} \R^N$ with $\pp_{y_N}u >0$ must be such that its level sets $\{u=\la\}$ are all hyperplanes, {\em \bf at least} for dimension $N\le 8$. A counterexample for $N\ge 9$ has long been believed to exist. …

This paper introduces Non-Autonomous Input-Output Stable Network(NAIS-Net), a very deep architecture where each stacked processing block is derived from a time-invariant non-autonomous dynamical system. Non-autonomy is implemented by skip connections from the block input to each of the unrolled processing stages and al…

The paper identifies a new geometric and spectral phenomenon in the critical hyperbolic catenoid family.

In a previous paper we obtained formulae for the volume of a causal diamond or Alexandrov open set $I^+(p) \cap I^-(q)$ whose duration $τ(p,q)$ is short compared with the curvature scale. In the present paper we obtain asymptotic formulae valid when the point $q$ recedes to the future boundary ${\cal I}^+$ of an asymp…

Sigmoid-type networks avoid vanishing gradients with regularization and rescaling.

Global convergence of SGD proven for two-layer neural nets with regularization.

FleXOR trains fractional quantization for neural networks, improving accuracy and size.

In this work, we propose a novel recurrent neural network (RNN) architecture. The proposed RNN, gated-feedback RNN (GF-RNN), extends the existing approach of stacking multiple recurrent layers by allowing and controlling signals flowing from upper recurrent layers to lower layers using a global gating unit for each pai…

Using back-propagation and its variants to train deep networks is often problematic for new users. Issues such as exploding gradients, vanishing gradients, and high sensitivity to weight initialization strategies often make networks difficult to train, especially when users are experimenting with new architectures. Her…

In this paper, we consider parameter recovery for non-overlapping convolutional neural networks (CNNs) with multiple kernels. We show that when the inputs follow Gaussian distribution and the sample size is sufficiently large, the squared loss of such CNNs is $\mathit{~locally~strongly~convex}$ in a basin of attraction…

Exact bounds derived for neural network outputs with noisy inputs.

Paper establishes bounds for RNN-TPPs, showing four-layer networks can achieve vanishing errors.

Study shows DNNs can recover functions with fewer samples than model parameters at overparameterization.

The performance of artificial neural networks (ANNs) is influenced by weight initialization, the nature of activation functions, and their architecture. There is a wide range of activation functions that are traditionally used to train a neural network, e.g. sigmoid, tanh, and Rectified Linear Unit (ReLU). A widespread…

We propose $\textit{Mish}$, a novel self-regularized non-monotonic activation function which can be mathematically defined as: $f(x)=x\tanh(softplus(x))$. As activation functions play a crucial role in the performance and training dynamics in neural networks, we validated experimentally on several well-known benchmarks…

A complex-valued convolutional network (convnet) implements the repeated application of the following composition of three operations, recursively applying the composition to an input vector of nonnegative real numbers: (1) convolution with complex-valued vectors followed by (2) taking the absolute value of every entry…

The paper proves consistency of neural networks with regularization.

Efficiently prices American options with multiple assets using sparse grids.