arXiv research

ResNet learns to compress information during training.

Adversarial attack methods have demonstrated the fragility of deep neural networks. Their imperceptible perturbations are frequently able fool classifiers into potentially dangerous misclassifications. We propose a novel way to interpret adversarial perturbations in terms of the effective input signal that classifiers …

The method learns disentangled representations for localized image manipulations.

Residual networks (ResNets) have recently achieved state-of-the-art on challenging computer vision tasks. We introduce Resnet in Resnet (RiR): a deep dual-stream architecture that generalizes ResNets and standard CNNs and is easily implemented with no computational overhead. RiR consistently improves performance over R…

Momentum ResNets improve ResNets' memory efficiency.

Stable ResNet stabilizes gradients in deep networks.

Paper interprets ResNets via gate-network controls and deep-layer classifications.

ResNets and their GP generalization align ideas via function space warping, revealing robust properties and connections to image registration.

Deep residual networks (ResNets) and their variants are widely used in many computer vision applications and natural language processing tasks. However, the theoretical principles for designing and training ResNets are still not fully understood. Recently, several points of view have emerged to try to interpret ResNet …

ResNets learn the geodesic curve in Wasserstein space.

Universal approximation for ODENet and ResNet with a single activation function.

Residual Network (ResNet) is the state-of-the-art architecture that realizes successful training of really deep neural network. It is also known that good weight initialization of neural network avoids problem of vanishing/exploding gradients. In this paper, simplified models of ResNets are analyzed. We argue that good…

We present a numerical framework for approximating unknown governing equations using observation data and deep neural networks (DNN). In particular, we propose to use residual network (ResNet) as the basic building block for equation approximation. We demonstrate that the ResNet block can be considered as a one-step me…

Deep ResNets favor low bottleneck rank with proper hyperparameters.

We study the stability and convergence of training deep ResNets with gradient descent. Specifically, we show that the parametric branch in the residual block should be scaled down by a factor $τ=O(1/\sqrt{L})$ to guarantee stable forward/backward process, where $L$ is the number of residual blocks. Moreover, we establi…

ResNets promote smoother interpolations than MLPs, enhancing generalization.

In this paper, we aim to understand Residual Network (ResNet) in a scientifically sound way by providing a bridge between ResNet and Feynman path integral. In particular, we prove that the effect of residual block is equivalent to partial differential equation, and the ResNet transforming process can be equivalently co…

Recent results in the literature indicate that a residual network (ResNet) composed of a single residual block outperforms linear predictors, in the sense that all local minima in its optimization landscape are at least as good as the best linear predictor. However, these results are limited to a single residual block …

ResNets can approximate arbitrary ODEs in space and time.

ResNets minimize circuit size for fitting data in HTMC regime.

Gradient descent finds global optima in ResNets with sufficient parameters.

Residual Neural Networks (ResNets) achieve state-of-the-art performance in many computer vision problems. Compared to plain networks without residual connections (PlnNets), ResNets train faster, generalize better, and suffer less from the so-called degradation problem. We introduce simplified (but still nonlinear) vers…

ResNets approximate log-Gaussian at initialization, improving network performance.

Neural ODEs and i-ResNets can't approximate all continuous invertible functions.

Neural Network is a powerful Machine Learning tool that shows outstanding performance in Computer Vision, Natural Language Processing, and Artificial Intelligence. In particular, recently proposed ResNet architecture and its modifications produce state-of-the-art results in image classification problems. ResNet and mos…

This paper extends ResNet theory to infinitely deep networks, linking them to diffusion processes.

Generalized ResNet learns unknown dynamical systems using neural networks.

We investigate whether ResNet architectures can outperform more traditional Convolutional Neural Networks on the task of fine-grained vehicle classification. We train and test ResNet-18, ResNet-34 and ResNet-50 on the Comprehensive Cars dataset without pre-training on other datasets. We then modify the networks to use …

Non-determinism from GPUs dominates ResNet training accuracy variability.

New method trains deep vanilla networks as fast as ResNets without shortcut connections.

This paper analyzes Stochastic Depth regularization in ResNets.

This paper explains why ResNets generalize better than FFNets using neural tangent kernels.

We demonstrate that a very deep ResNet with stacked modules with one neuron per hidden layer and ReLU activation functions can uniformly approximate any Lebesgue integrable function in $d$ dimensions, i.e. $\ell_1(\mathbb{R}^d)$. Because of the identity mapping inherent to ResNets, our network has alternating layers of…

In our work, we bridge deep neural network design with numerical differential equations. We show that many effective networks, such as ResNet, PolyNet, FractalNet and RevNet, can be interpreted as different numerical discretizations of differential equations. This finding brings us a brand new perspective on the design…

Skip connections make adversarial examples more transferable in ResNets.

Paper studies ResNet dynamics using NTH, reducing width requirement.

Residual Networks (ResNets) have become state-of-the-art models in deep learning and several theoretical studies have been devoted to understanding why ResNet works so well. One attractive viewpoint on ResNet is that it is optimizing the risk in a functional space by combining an ensemble of effective features. In this…

Shortcut connections in ResNet help avoid local optima, leading to efficient training.

Hybrid ResNet and RMT improve covariance matrix estimation for cryptocurrency portfolios.

One of the ways to train deep neural networks effectively is to use residual connections. Residual connections can be classified as being either identity connections or bridge-connections with a reshaping convolution. Empirical observations on CIFAR-10 and CIFAR-100 datasets using a baseline Resnet model, with bridge-c…

Deep ResNets can achieve zero loss with enough layers and weights.

In this work we apply variations of ResNet architecture to the task of atrial fibrillation classification. Variations differ in number of filter after first convolution, ResNet block layout, number of filters in block convolutions and number of ResNet blocks between downsampling operations. We have found a range of mod…

An adaptive time-stepping controller improves stability and accuracy of ResNets.

ResNets and DenseNets converge to NTK with depth and width, offering advantages for kernel regression.

New method improves ResNet performance without batch normalization.

Empirical adversarial risk minimization (EARM) is a widely used mathematical framework to robustly train deep neural nets (DNNs) that are resistant to adversarial attacks. However, both natural and robust accuracies, in classifying clean and adversarial images, respectively, of the trained robust models are far from sa…

Study on feature learning in Leaky ResNets, explaining bottleneck structure.

UDENet and ResNet can approximate any function, with ODENet showing UAP for continuous functions.