arXiv research

Paper interprets ResNets via gate-network controls and deep-layer classifications.

Stable ResNet stabilizes gradients in deep networks.

Deep ResNets favor low bottleneck rank with proper hyperparameters.

Residual Network (ResNet) is the state-of-the-art architecture that realizes successful training of really deep neural network. It is also known that good weight initialization of neural network avoids problem of vanishing/exploding gradients. In this paper, simplified models of ResNets are analyzed. We argue that good…

ResNets learn the geodesic curve in Wasserstein space.

Deep residual networks (ResNets) have demonstrated better generalization performance than deep feedforward networks (FFNets). However, the theory behind such a phenomenon is still largely unknown. This paper studies this fundamental problem in deep learning from a so-called "neural tangent kernel" perspective. Specific…

We study the stability and convergence of training deep ResNets with gradient descent. Specifically, we show that the parametric branch in the residual block should be scaled down by a factor $τ=O(1/\sqrt{L})$ to guarantee stable forward/backward process, where $L$ is the number of residual blocks. Moreover, we establi…

New method trains deep vanilla networks as fast as ResNets without shortcut connections.

Recent results in the literature indicate that a residual network (ResNet) composed of a single residual block outperforms linear predictors, in the sense that all local minima in its optimization landscape are at least as good as the best linear predictor. However, these results are limited to a single residual block …

Residual networks (ResNets) have recently achieved state-of-the-art on challenging computer vision tasks. We introduce Resnet in Resnet (RiR): a deep dual-stream architecture that generalizes ResNets and standard CNNs and is easily implemented with no computational overhead. RiR consistently improves performance over R…

ResNets promote smoother interpolations than MLPs, enhancing generalization.

Universal approximation for ODENet and ResNet with a single activation function.

Momentum ResNets improve ResNets' memory efficiency.

This paper extends ResNet theory to infinitely deep networks, linking them to diffusion processes.

In this paper we prove that, in the deep limit, the stochastic gradient descent on a ResNet type deep neural network, where each layer shares the same weight matrix, converges to the stochastic gradient descent for a Neural ODE and that the corresponding value/loss functions converge. Our result gives, in the context o…

We present a numerical framework for approximating unknown governing equations using observation data and deep neural networks (DNN). In particular, we propose to use residual network (ResNet) as the basic building block for equation approximation. We demonstrate that the ResNet block can be considered as a one-step me…

Deep ResNets can achieve zero loss with enough layers and weights.

We demonstrate that a very deep ResNet with stacked modules with one neuron per hidden layer and ReLU activation functions can uniformly approximate any Lebesgue integrable function in $d$ dimensions, i.e. $\ell_1(\mathbb{R}^d)$. Because of the identity mapping inherent to ResNets, our network has alternating layers of…

In our work, we bridge deep neural network design with numerical differential equations. We show that many effective networks, such as ResNet, PolyNet, FractalNet and RevNet, can be interpreted as different numerical discretizations of differential equations. This finding brings us a brand new perspective on the design…

ResNet learns to compress information during training.

ResNets minimize circuit size for fitting data in HTMC regime.

Deep residual networks (ResNets) and their variants are widely used in many computer vision applications and natural language processing tasks. However, the theoretical principles for designing and training ResNets are still not fully understood. Recently, several points of view have emerged to try to interpret ResNet …

Deep ResNets exhibit distinct scaling properties with depth, challenging neural ODE models.

UDENet and ResNet can approximate any function, with ODENet showing UAP for continuous functions.

A new multilevel framework speeds up ResNet training.

We present a general numerical approach for learning unknown dynamical systems using deep neural networks (DNNs). Our method is built upon recent studies that identified the residue network (ResNet) as an effective neural network structure. In this paper, we present a generalized ResNet framework and broadly define res…

Paper studies ResNet dynamics using NTH, reducing width requirement.

Residual networks (ResNets) are a deep learning architecture that substantially improved the state of the art performance in certain supervised learning tasks. Since then, they have received continuously growing attention. ResNets have a recursive structure $x_{k+1} = x_k + R_k(x_k)$ where $R_k$ is a neural network cal…

Our research proves neural collapse in deep ResNets and transformers is globally optimal.

Gradient amplification boosts deep learning model performance without increasing training time.

In this paper, we prove that depth with nonlinearity creates no bad local minima in a type of arbitrarily deep ResNets with arbitrary nonlinear activation functions, in the sense that the values of all local minima are no worse than the global minimum value of corresponding classical machine-learning models, and are gu…

CPOT prunes deep networks by identifying redundant filters using optimal transport.

PruneNet efficiently prunes channels in deep networks, improving accuracy and performance.

Unified scheme combining softmax and ResNet for deep learning.

Scaling ResNets requires careful consideration of the layer depth and output scaling factors.

This paper analyzes Stochastic Depth regularization in ResNets.

Dropout and RaM become equivalent in large ResNets as depth and width increase.

PETRA enables parallel training of deep models with reversible architectures.

We study the convergence of gradient descent (GD) and stochastic gradient descent (SGD) for training $L$-hidden-layer linear residual networks (ResNets). We prove that for training deep residual networks with certain linear transformations at input and output layers, which are fixed throughout training, both GD and SGD…

Residual Networks (ResNets) have become state-of-the-art models in deep learning and several theoretical studies have been devoted to understanding why ResNet works so well. One attractive viewpoint on ResNet is that it is optimizing the risk in a functional space by combining an ensemble of effective features. In this…

The paper analyzes deep ReLU CNNs' approximation properties in 2D space.

ResNets approximate log-Gaussian at initialization, improving network performance.

One of the ways to train deep neural networks effectively is to use residual connections. Residual connections can be classified as being either identity connections or bridge-connections with a reshaping convolution. Empirical observations on CIFAR-10 and CIFAR-100 datasets using a baseline Resnet model, with bridge-c…

New method improves ResNet performance without batch normalization.

Develops EFT for ResNets, revealing limitations of kernel-only approach.

Researchers extend ResNets to Riemannian manifolds, improving performance over existing methods.

New method trains deep ResNets without normalization, achieving state-of-the-art performance.

We show that the stochasticity in training ResNets for image classification on GPUs in TensorFlow is dominated by the non-determinism from GPUs, rather than by the initialisation of the weights and biases of the network or by the sequence of minibatches given. The standard deviation of test set accuracy is 0.02 with fi…