arXiv research

Residual neural networks don't help overcome sampling complexity issues.

Generalization bounds derived for neural ODEs and deep residual networks.

Proposes a neural network method to correct residual distortions in coordinate transformations.

Deep residual networks implicitly converge to neural ODEs.

Residual connections significantly boost the performance of deep neural networks. However, there are few theoretical results that address the influence of residuals on the hypothesis complexity and the generalization ability of deep neural networks. This paper studies the influence of residual connections on the hypoth…

Wide residual networks generalize well with uniform convergence to RNTK as width increases.

Graph neural networks suffer from oversmoothing, but adding residual connections helps.

Residual networks with block width max(d_x, d_y) approximate all functions.

Deep residual networks can approximate any continuous function using control theory.

Review of neural network expressivity and architectures.

Adaptive weights improve physics-informed neural networks and deep operator networks.

Investigates the relationship between ResNets and Neural ODEs, quantifying their closeness and providing training methods.

Speeds up deep neural networks training by 10x using GPU concurrency.

Residual generation helps diagnose engine faults using neural networks.

GT-DDP optimizer trains residual networks using game theory.

Networks with large receptive field (RF) have shown advanced fitting ability in recent years. In this work, we utilize the short-term residual learning method to improve the performance and robustness of networks for image denoising tasks. Here, we choose a multi-wavelet convolutional neural network (MWCNN), one of the…

ResGCN detects anomalies in attributed networks by capturing sparsity and nonlinearity.

Unified ODE model explains residual and non-residual networks.

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

Study shows how deep residual networks can be analyzed as shallow network ensembles for optimization.

In this paper, we introduce transformations of deep rectifier networks, enabling the conversion of deep rectifier networks into shallow rectifier networks. We subsequently prove that any rectifier net of any depth can be represented by a maximum of a number of functions that can be realized by a shallow network with a …

We introduce a new deep convolutional neural network, CrescendoNet, by stacking simple building blocks without residual connections. Each Crescendo block contains independent convolution paths with increased depths. The numbers of convolution layers and parameters are only increased linearly in Crescendo blocks. In exp…

Gradient descent finds a global minimum in training deep neural networks despite the objective function being non-convex. The current paper proves gradient descent achieves zero training loss in polynomial time for a deep over-parameterized neural network with residual connections (ResNet). Our analysis relies on the p…

ResMem improves model generalization by explicitly memorizing residuals.

To have a superior generalization, a deep learning neural network often involves a large size of training sample. With increase of hidden layers in order to increase learning ability, neural network has potential degradation in accuracy. Both could seriously limit applicability of deep learning in some domains particul…

New measure assesses deep neural networks' robustness to adversarial attacks.

ResNets promote smoother interpolations than MLPs, enhancing generalization.

The skip-connections used in residual networks have become a standard architecture choice in deep learning due to the increased training stability and generalization performance with this architecture, although there has been limited theoretical understanding for this improvement. In this work, we analyze overparameter…

Researchers extend ResNets to Riemannian manifolds, improving performance over existing methods.

Residual Network (ResNet) is the state-of-the-art architecture that realizes successful training of really deep neural network. It is also known that good weight initialization of neural network avoids problem of vanishing/exploding gradients. In this paper, simplified models of ResNets are analyzed. We argue that good…

The paper establishes principles for initializing and designing GNNs with ReLU activations to avoid oversmoothing and correlation collapse.

AAS optimizes neural network PDE approximations by adaptively sampling.

JKO-iFlow uses neural ODEs to improve generative models with reduced memory and training complexity.

Deep, wide ConvResNets can approximate functions and their smoothness.

STRIC detects anomalies in time series by analyzing residual signals.

While training error of most deep neural networks degrades as the depth of the network increases, residual networks appear to be an exception. We show that the main reason for this is the Lyapunov stability of the gradient descent algorithm: for an arbitrarily chosen step size, the equilibria of the gradient descent ar…

Stacking improves deep neural network training efficiency.

We present a new physics informed neural network (PINN) algorithm for solving brittle fracture problems. While most of the PINN algorithms available in the literature minimize the residual of the governing partial differential equation, the proposed approach takes a different path by minimizing the variational energy o…

Residual networks (ResNets) are a deep learning architecture that substantially improved the state of the art performance in certain supervised learning tasks. Since then, they have received continuously growing attention. ResNets have a recursive structure $x_{k+1} = x_k + R_k(x_k)$ where $R_k$ is a neural network cal…

Momentum ResNets improve ResNets' memory efficiency.

SRFRN accelerates image super-resolution using shallow residual units.

Deep learning enhances options hedging performance.

RFRBoost uses random features to boost deep residual neural networks, improving performance and computational efficiency.

Hybrid method improves SABR implied volatility approximation.

New bounds on ReLU networks for low-regular functions.

RR-GNN improves GNN prediction intervals by accounting for graph heteroscedasticity and structural biases.

New method diagnoses criticality in deep neural networks, improving performance.

Recently, deep residual networks have been successfully applied in many computer vision and natural language processing tasks, pushing the state-of-the-art performance with deeper and wider architectures. In this work, we interpret deep residual networks as ordinary differential equations (ODEs), which have long been s…