arXiv research

Wide residual networks generalize well with uniform convergence to RNTK as width increases.

Batch normalization makes deep residual networks train faster.

Unified ODE model explains residual and non-residual networks.

ResGCN detects anomalies in attributed networks by capturing sparsity and nonlinearity.

Residual neural networks don't help overcome sampling complexity issues.

The skip-connections used in residual networks have become a standard architecture choice in deep learning due to the increased training stability and generalization performance with this architecture, although there has been limited theoretical understanding for this improvement. In this work, we analyze overparameter…

Proposes a neural network method to correct residual distortions in coordinate transformations.

This paper studies deep learning methodologies for portfolio optimization in the US equities market. We present a novel residual switching network that can automatically sense changes in market regimes and switch between momentum and reversal predictors accordingly. The residual switching network architecture combines …

Enhanced image denoising with MWRDCNN using residual dense blocks.

We conduct mathematical analysis on the effect of batch normalization (BN) on gradient backpropogation in residual network training, which is believed to play a critical role in addressing the gradient vanishing/explosion problem, in this work. By analyzing the mean and variance behavior of the input and the gradient i…

Generalization bounds derived for neural ODEs and deep residual networks.

SRFRN accelerates image super-resolution using shallow residual units.

Deep residual networks implicitly converge to neural ODEs.

Residual Continual Learning prevents forgetting in sequential tasks.

Residual networks with block width max(d_x, d_y) approximate all functions.

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

Deep residual networks can approximate any continuous function using control theory.

Residual connections significantly boost the performance of deep neural networks. However, there are few theoretical results that address the influence of residuals on the hypothesis complexity and the generalization ability of deep neural networks. This paper studies the influence of residual connections on the hypoth…

RDL-Net improves speech enhancement with fewer parameters and better performance.

Graph neural networks suffer from oversmoothing, but adding residual connections helps.

Study shows how deep residual networks can be analyzed as shallow network ensembles for optimization.

While training error of most deep neural networks degrades as the depth of the network increases, residual networks appear to be an exception. We show that the main reason for this is the Lyapunov stability of the gradient descent algorithm: for an arbitrarily chosen step size, the equilibria of the gradient descent ar…

Novel LSTM network predicts pulsar timing residuals with few-shot data.

Residual networks' depth is mathematically equivalent to expanding an implicit ensemble size.

We revisit residual algorithms in both model-free and model-based reinforcement learning settings. We propose the bidirectional target network technique to stabilize residual algorithms, yielding a residual version of DDPG that significantly outperforms vanilla DDPG in the DeepMind Control Suite benchmark. Moreover, we…

Adaptive weights improve physics-informed neural networks and deep operator networks.

We regard pre-trained residual networks (ResNets) as nonlinear systems and use linearization, a common method used in the qualitative analysis of nonlinear systems, to understand the behavior of the networks under small perturbations of the input images. We work with ResNet-56 and ResNet-110 trained on the CIFAR-10 dat…

Flow-based generative models parameterize probability distributions through an invertible transformation and can be trained by maximum likelihood. Invertible residual networks provide a flexible family of transformations where only Lipschitz conditions rather than strict architectural constraints are needed for enforci…

GT-DDP optimizer trains residual networks using game theory.

An emerging design principle in deep learning is that each layer of a deep artificial neural network should be able to easily express the identity transformation. This idea not only motivated various normalization techniques, such as \emph{batch normalization}, but was also key to the immense success of \emph{residual …

Extends VAEs to handle complex Bayesian network structures.

Invertible DenseNets improve model efficiency and performance.

In this paper, we introduce transformations of deep rectifier networks, enabling the conversion of deep rectifier networks into shallow rectifier networks. We subsequently prove that any rectifier net of any depth can be represented by a maximum of a number of functions that can be realized by a shallow network with a …

Residual generation helps diagnose engine faults using neural networks.

ResMem improves model generalization by explicitly memorizing residuals.

Investigates the relationship between ResNets and Neural ODEs, quantifying their closeness and providing training methods.

We show that Residual Networks (ResNet) is equivalent to boosting feature representation, without any modification to the underlying ResNet training algorithm. A regret bound based on Online Gradient Boosting theory is proved and suggests that ResNet could achieve Online Gradient Boosting regret bounds through neural n…

Inverse modeling for the estimation of non-Gaussian hydraulic conductivity fields in subsurface flow and solute transport models remains a challenging problem. This is mainly due to the non-Gaussian property, the non-linear physics, and the fact that many repeated evaluations of the forward model are often required. In…

We analyze the global convergence of gradient descent for deep linear residual networks by proposing a new initialization: zero-asymmetric (ZAS) initialization. It is motivated by avoiding stable manifolds of saddle points. We prove that under the ZAS initialization, for an arbitrary target matrix, gradient descent con…

Theoretical limits of deep residual networks show consistent covariance structures.

To have a superior generalization, a deep learning neural network often involves a large size of training sample. With increase of hidden layers in order to increase learning ability, neural network has potential degradation in accuracy. Both could seriously limit applicability of deep learning in some domains particul…

Review of neural network expressivity and architectures.

MLP residual networks implement a selective coarse-graining procedure governed by the spectral structure of the input distribution.

Speeds up deep neural networks training by 10x using GPU concurrency.

STRIC detects anomalies in time series by analyzing residual signals.

Accelerated magnetic resonance (MR) scan acquisition with compressed sensing (CS) and parallel imaging is a powerful method to reduce MR imaging scan time. However, many reconstruction algorithms have high computational costs. To address this, we investigate deep residual learning networks to remove aliasing artifacts …

In this paper we propose the use of continuous residual modules for graph kernels in Graph Neural Networks. We show how both discrete and continuous residual layers allow for more robust training, being that continuous residual layers are those which are applied by integrating through an Ordinary Differential Equation …

ResNets promote smoother interpolations than MLPs, enhancing generalization.