arXiv research

Wide residual networks generalize well with uniform convergence to RNTK as width increases.

Batch normalization makes deep residual networks train faster.

Residual neural networks don't help overcome sampling complexity issues.

This paper proposes a deep speech enhancement method which exploits the high potential of residual connections in a wide neural network architecture, a topology known as Wide Residual Network. This is supported on single dimensional convolutions computed alongside the time domain, which is a powerful approach to proces…

Deep, wide ConvResNets can approximate functions and their smoothness.

Deep residual networks implicitly converge to neural ODEs.

Improved DNN calibration without sacrificing accuracy.

Highly distributed training of Deep Neural Networks (DNNs) on future compute platforms (offering 100 of TeraOps/s of computational capacity) is expected to be severely communication constrained. To overcome this limitation, new gradient compression techniques are needed that are computationally friendly, applicable to …

Deep linear ResNets converge globally with certain transformations.

Study proposes a new method for deep portfolio optimization using residual factors.

Deep residual networks (ResNets) and their variants are widely used in many computer vision applications and natural language processing tasks. However, the theoretical principles for designing and training ResNets are still not fully understood. Recently, several points of view have emerged to try to interpret ResNet …

Normalization layers are a staple in state-of-the-art deep neural network architectures. They are widely believed to stabilize training, enable higher learning rate, accelerate convergence and improve generalization, though the reason for their effectiveness is still an active research topic. In this work, we challenge…

Stacking improves deep neural network training efficiency.

SMART-FAN-Lasso fine-tunes neural networks for high-dimensional nonparametric regression.

New measure assesses deep neural networks' robustness to adversarial attacks.

Convolutional Neural Networks (CNNs) filter the input data using spatial convolution operators with compact stencils. Commonly, the convolution operators couple features from all channels, which leads to immense computational cost in the training of and prediction with CNNs. To improve the efficiency of CNNs, we introd…

Wide Bayesian neural networks have a simpler weight posterior, leading to faster MCMC sampling.

Unified learning-rate scale for CNNs and ResNets, avoiding depth imbalance.

Wide hidden layer TCM nets capacity analyzed using RDT and fl RDT.

Improved ResNets and DenseNets models for better feature reuse.

Unified ODE model explains residual and non-residual networks.

ResGCN detects anomalies in attributed networks by capturing sparsity and nonlinearity.

Residual Networks with convolutional layers are widely used in the field of machine learning. Since they effectively extract features from input data by stacking multiple layers, they can achieve high accuracy in many applications. However, the stacking of many layers raises their computation costs. To address this pro…

New method diagnoses criticality in deep neural networks, improving performance.

Analyzes dynamics of quantum neural networks, predicting exponential decay of training error.

Wide neural networks learn features under $μ$P, identifying weights and decomposing support.

The skip-connections used in residual networks have become a standard architecture choice in deep learning due to the increased training stability and generalization performance with this architecture, although there has been limited theoretical understanding for this improvement. In this work, we analyze overparameter…

Effective theory for Transformer initialization improves model performance.

GCNIII combines Wide & Deep for better node classification.

Time-aware deep learning methods improve spatial downscaling of atmospheric pollutants.

Ensemble methods, traditionally built with independently trained de-correlated models, have proven to be efficient methods for reducing the remaining residual generalization error, which results in robust and accurate methods for real-world applications. In the context of deep learning, however, training an ensemble of…

Proposes a neural network method to correct residual distortions in coordinate transformations.

Due to the rapid growth of machine learning tools and specifically deep networks in various computer vision and image processing areas, application of Convolutional Neural Networks for watermarking have recently emerged. In this paper, we propose a deep end-to-end diffusion watermarking framework (ReDMark) which can be…

A new algorithm estimates output ranges for deep neural networks efficiently.

This paper studies deep learning methodologies for portfolio optimization in the US equities market. We present a novel residual switching network that can automatically sense changes in market regimes and switch between momentum and reversal predictors accordingly. The residual switching network architecture combines …

We conduct mathematical analysis on the effect of batch normalization (BN) on gradient backpropogation in residual network training, which is believed to play a critical role in addressing the gradient vanishing/explosion problem, in this work. By analyzing the mean and variance behavior of the input and the gradient i…

Generalization bounds derived for neural ODEs and deep residual networks.

SRFRN accelerates image super-resolution using shallow residual units.

Residual networks with block width max(d_x, d_y) approximate all functions.

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

Deep residual networks can approximate any continuous function using control theory.

Residual connections significantly boost the performance of deep neural networks. However, there are few theoretical results that address the influence of residuals on the hypothesis complexity and the generalization ability of deep neural networks. This paper studies the influence of residual connections on the hypoth…

RDL-Net improves speech enhancement with fewer parameters and better performance.

Graph neural networks suffer from oversmoothing, but adding residual connections helps.

Study shows how deep residual networks can be analyzed as shallow network ensembles for optimization.

This paper extends ResNet theory to infinitely deep networks, linking them to diffusion processes.

For fast and energy-efficient deployment of trained deep neural networks on resource-constrained embedded hardware, each learned weight parameter should ideally be represented and stored using a single bit. Error-rates usually increase when this requirement is imposed. Here, we report large improvements in error rates …

While training error of most deep neural networks degrades as the depth of the network increases, residual networks appear to be an exception. We show that the main reason for this is the Lyapunov stability of the gradient descent algorithm: for an arbitrarily chosen step size, the equilibria of the gradient descent ar…