arXiv research

Randomly trained neural networks can generalize well if there's a simpler underlying teacher model.

The paper trains neural networks with robustness guarantees using semidefinite constraints.

Study on dropout in neural networks using percolation theory.

New MIP methods improve training of integer-valued neural networks.

A fast method for LOOCV in k-NN regression reduces computation time.

Training a Neural Network (NN) with lots of parameters or intricate architectures creates undesired phenomena that complicate the optimization process. To address this issue we propose a first modular approach to NN design, wherein the NN is decomposed into a control module and several functional modules, implementing …

The paper proposes a method to train NNs with a small Lipschitz constant to improve robustness.

Large learning rates cause oscillations in NN weights that improve generalization.

NNs accurately predict energy eigenvalues and other physical phenomena in 1D quantum mechanics.

This work establishes the equivalence between neural networks and support vector machines.

This paper studies large-width asymptotics for ReLU neural networks with α-Stable initializations.

New method trains quantized neural networks to global optimality.

AOPU stabilizes NN training by approximating natural gradient, improving stability and convergence.

Improves neural network performance by enriching training dataset.

The neural network (NN)-based direct uncertainty quantification (UQ) methods have achieved the state of the art performance since the first inauguration, known as the lower-upper-bound estimation (LUBE) method. However, currently-available cost functions for uncertainty guided NN training are not always converging and …

With a constant improvement in the network architectures and training methodologies, Neural Networks (NNs) are increasingly being deployed in real-world Machine Learning systems. However, despite their impressive performance on "known inputs", these NNs can fail absurdly on the "unseen inputs", especially if these real…

For neural networks (NNs) with rectified linear unit (ReLU) or binary activation functions, we show that their training can be accomplished in a reduced parameter space. Specifically, the weights in each neuron can be trained on the unit sphere, as opposed to the entire space, and the threshold can be trained in a boun…

Neural Networks (NNs) have been extensively used for a wide spectrum of real-world regression tasks, where the goal is to predict a numerical outcome such as revenue, effectiveness, or a quantitative result. In many such tasks, the point prediction is not enough: the uncertainty (i.e. risk or confidence) of that predic…

A current research trend in neurocomputing involves the design of novel artificial neural networks incorporating the concept of time into their operating model. In this paper, a novel architecture that employs stigmergy is proposed. Computational stigmergy is used to dynamically increase (or decrease) the strength of a…

We focus on estimating \emph{a priori} generalization error of two-layer ReLU neural networks (NNs) trained by mean squared error, which only depends on initial parameters and the target function, through the following research line. We first estimate \emph{a priori} generalization error of finite-width two-layer ReLU …

This study explains how different training methods affect the minimizer of neural networks.

SGD trains NNs to learn low-dimensional representations efficiently.

We present NN-grams, a novel, hybrid language model integrating n-grams and neural networks (NN) for speech recognition. The model takes as input both word histories as well as n-gram counts. Thus, it combines the memorization capacity and scalability of an n-gram model with the generalization ability of neural network…

Simple k-NN filtering improves model accuracy on noisy labels.

New method protects neural networks from adversarial attacks without generating adversarial examples.

Overparameterized Neural Networks (NN) display state-of-the-art performance. However, there is a growing need for smaller, energy-efficient, neural networks tobe able to use machine learning applications on devices with limited computational resources. A popular approach consists of using pruning techniques. While thes…

Develops deep learning for optimizing 5G radio resource allocation.

Neural network discovers exact solutions to QP with linear constraints.

This paper extends stability and generalization analysis of GD for multi-layer NNs.

Bayesian neural networks with functional priors improve surrogate modeling in mechanics.

Optimizes neural networks for solving problems with pruning and ensembles of minimal structures.

Improved neural network models predict molecular and material properties efficiently.

Different neural network (NN) architectures have different advantages. Convolutional neural networks (CNNs) achieved enormous success in computer vision, while recurrent neural networks (RNNs) gained popularity in speech recognition. It is not known which type of NN architecture is the best fit for classification of co…

Ensembles of neural networks (NNs) have long been used to estimate predictive uncertainty; a small number of NNs are trained from different initialisations and sometimes on differing versions of the dataset. The variance of the ensemble's predictions is interpreted as its epistemic uncertainty. The appeal of ensembling…

Noise injection (NI) is an efficient technique to mitigate over-fitting in neural networks (NNs). The Bernoulli NI procedure as implemented in dropout and shakeout has connections with $l_1$ and $l_2$ regularization for the NN model parameters. We propose whiteout, a family NI regularization techniques (NIRT) through i…

The study analyzes a three-layer neural network's training dynamics using a functional-space mean-field theory.

In many fields such as bioinformatics, high energy physics, power distribution, etc., it is desirable to learn non-linear models where a small number of variables are selected and the interaction between them is explicitly modeled to predict the response. In principle, neural networks (NNs) could accomplish this task s…

We describe tests validating progress made toward acceleration and automation of hydrodynamic codes in the regime of developed turbulence by three Deep Learning (DL) Neural Network (NN) schemes trained on Direct Numerical Simulations of turbulence. Even the bare DL solutions, which do not take into account any physics …

This work defines a new function space for multi-layer neural networks.

The intrinsic error tolerance of neural network (NN) makes approximate computing a promising technique to improve the energy efficiency of NN inference. Conventional approximate computing focuses on balancing the efficiency-accuracy trade-off for existing pre-trained networks, which can lead to suboptimal solutions. In…

This paper analyzes deep Stable neural networks, showing convergence rates under different growth settings.

We prove overfitting in minimal and random NNs, tempering the effect.

The paper characterizes functions of shallow ReLU NN denoisers under minimal norm constraints.

SynergicLearning combines NN and HD models for high accuracy and efficiency.

Bayesian deep ensembles improve prediction accuracy in various settings.

Deep Reinforcement Learning (RL) demonstrates excellent performance on tasks that can be solved by trained policy. It plays a dominant role among cutting-edge machine learning approaches using multi-layer Neural networks (NNs). At the same time, Deep RL suffers from high sensitivity to noisy, incomplete, and misleading…

This paper studies the relationship between the classification performed by deep neural networks (DNNs) and the decision of various classical classifiers, namely k-nearest neighbours (k-NN), support vector machines (SVM) and logistic regression (LR), at various layers of the network. This comparison provides us with ne…

Paper optimizes neural network initialization using SMT solvers.