Wide residual networks generalize well with uniform convergence to RNTK as width increases.
problem Understanding the generalization ability of wide residual networks.
method Uniform convergence of residual network kernel to residual neural tangent kernel (RNTK).
result Generalization error converges to kernel regression error with respect to RNTK.
Batch normalization makes deep residual networks train faster.
problem Training deep residual networks with large depths.
method Downscaling the residual branch by a normalizing factor early in training.
result Normalized residual blocks compute functions close to the identity function early in training.
This paper proposes a deep speech enhancement method which exploits the high potential of residual connections in a wide neural network architecture, a topology known as Wide Residual Network. This is supported on single dimensional convolutions computed alongside the time domain, which is a powerful approach to proces…
Residual neural networks don't help overcome sampling complexity issues.
problem Learning invertible residual neural networks from samples is hard due to the curse of dimensionality.
method Investigated invertible residual neural networks and their sampling complexity.
result Invertible residual neural networks still suffer from the curse of dimensionality in sampling complexity.
Deep, wide ConvResNets can approximate functions and their smoothness.
problem Function approximation and smoothness in deep networks.
method Analyzing ConvResNets, proving their ability to approximate functions and their smoothness.
result Large ConvResNets can approximate functions and exhibit sufficient first-order smoothness.
Deep residual networks implicitly converge to neural ODEs.
problem Link between discrete and continuous deep learning models.
method Establishing implicit regularization for residual networks towards neural ODEs.
result Deep residual networks initialized as discretizations of neural ODEs converge to such ODEs during training.
Improved DNN calibration without sacrificing accuracy.
problem Poor calibration of over-parametrized DNNs in safety-critical applications.
method Decoupling feature extraction and classification layers, and applying Gaussian priors.
result Significant improvement in model calibration with minimal training cost.
Highly distributed training of Deep Neural Networks (DNNs) on future compute platforms (offering 100 of TeraOps/s of computational capacity) is expected to be severely communication constrained. To overcome this limitation, new gradient compression techniques are needed that are computationally friendly, applicable to …
Deep linear ResNets converge globally with certain transformations.
problem Global convergence of training deep linear ResNets.
method Gradient descent and stochastic gradient descent for training L L L -hidden-layer linear ResNets. result GD and SGD can converge to global minimum for deep linear ResNets with specific transformations.
Study proposes a new method for deep portfolio optimization using residual factors.
problem Non-stationary financial market makes traditional machine learning methods ineffective.
method Predict distribution of residual factors using a novel neural network architecture with financial inductive biases.
result Demonstrated improved performance on U.S. and Japanese stock market data.
Deep residual networks (ResNets) and their variants are widely used in many computer vision applications and natural language processing tasks. However, the theoretical principles for designing and training ResNets are still not fully understood. Recently, several points of view have emerged to try to interpret ResNet …
Normalization layers are a staple in state-of-the-art deep neural network architectures. They are widely believed to stabilize training, enable higher learning rate, accelerate convergence and improve generalization, though the reason for their effectiveness is still an active research topic. In this work, we challenge…
Stacking improves deep neural network training efficiency.
problem Improving the efficiency of training deep neural networks.
method Proposes stacking as a form of accelerated gradient descent.
result Proves stacking provides accelerated training for certain deep linear residual networks.
SMART-FAN-Lasso fine-tunes neural networks for high-dimensional nonparametric regression.
problem Fine-tuning neural networks for high-dimensional nonparametric regression with variable selection.
method Source-model-augmented residual tuning (SMART) framework for neural Lasso.
result SMART-FAN-Lasso achieves statistical acceleration over single-task learning under precise conditions.
New measure assesses deep neural networks' robustness to adversarial attacks.
problem Deep learning's fragility to adversarial attacks limits its adoption in mission-critical applications.
method Introduces residual error as a new performance measure for assessing adversarial robustness.
result Demonstrates effectiveness of residual error in assessing robustness of deep neural networks.
Convolutional Neural Networks (CNNs) filter the input data using spatial convolution operators with compact stencils. Commonly, the convolution operators couple features from all channels, which leads to immense computational cost in the training of and prediction with CNNs. To improve the efficiency of CNNs, we introd…
Wide Bayesian neural networks have a simpler weight posterior, leading to faster MCMC sampling.
problem Sampling from the posterior of wide Bayesian neural networks is challenging.
method Introducing repriorisation, a data-dependent reparameterisation that simplifies the posterior distribution.
result The repriorisation map accelerates MCMC sampling, achieving up to 50x higher effective sample size.
Unified learning-rate scale for CNNs and ResNets, avoiding depth imbalance.
problem Challenges in choosing an appropriate learning rate for deep networks, especially as depth increases.
method Introduces Arithmetic-Mean μ μ μ P (AM- μ μ μ P), constraining network-wide average pre-activation second moment to a constant scale, combined with residual-aware He fan-in initialization. result Demonstrates a − 3 / 2 -3/2 − 3/2 scaling law for learning rates across depths, enabling zero-shot learning-rate transfer. Efficient facial feature learning with shared representations reduces redundancy and improves accuracy.
problem Redundancy and high computational load in training deep ensemble models.
method Wide Ensemble-based Convolutional Neural Networks (ESRs) with varying branching levels.
result ESRs reduce residual generalization error and outperform state-of-the-art methods on facial expression recognition.
Wide hidden layer TCM nets capacity analyzed using RDT and fl RDT.
problem Capacity analysis of wide hidden layer TCM nets.
method Employed Fully Lifted Random Duality Theory (fl RDT) for capacity characterization.
result Explicit, closed form capacity characterizations for a generic class of hidden layer activations.
Improved ResNets and DenseNets models for better feature reuse.
problem Diminishing feature reuse in ResNets and DenseNets.
method ResNEsts and DenseNEsts are block-based DNN models with improved representation guarantees.
result Wide ResNEsts with bottleneck blocks can guarantee desirable training properties.
Unified ODE model explains residual and non-residual networks.
problem Unclear relationship between residual and non-residual networks.
method Introducing a damping term in an ODE model to interpolate between ResNet and CNN architectures.
result Unified framework for understanding residual and non-residual networks.
Residual Networks with convolutional layers are widely used in the field of machine learning. Since they effectively extract features from input data by stacking multiple layers, they can achieve high accuracy in many applications. However, the stacking of many layers raises their computation costs. To address this pro…
New method diagnoses criticality in deep neural networks, improving performance.
problem Improving theoretical understanding and practical initialization of deep neural networks.
method Introducing partial Jacobians and deriving recurrence relations for their norms to analyze criticality.
result Proper stacking of LayerNorm and residual connections leads to a critical architecture for any initialization.
ResGCN detects anomalies in attributed networks by capturing sparsity and nonlinearity.
problem Detecting anomalous nodes in attributed networks.
method Attention-based deep residual modeling using Graph Convolutional Networks.
result ResGCN effectively detects anomalies in attributed networks.
Deep residual networks trained with gradient descent have small generalization gap.
problem Limited theoretical understanding of why residual networks generalize well.
method Analyzing overparameterized deep residual networks trained by gradient descent.
result Demonstrates that residual networks have a small generalization gap between training and test error.
Analyzes dynamics of quantum neural networks, predicting exponential decay of training error.
problem Understanding convergence rate of quantum neural networks training.
method Analytic theory for gradient descent dynamics of wide quantum neural networks.
result Simple analytic formula predicts exponential decay of training error.
Wide neural networks learn features under μ μ μ P, identifying weights and decomposing support.
problem Feature learning in wide neural networks under μ μ μ P. method Proving mean-field limit, characterizing identifiability, sparse-dictionary decomposition, and feature-learning-error decomposition.
result The triple ( w ∗ , D o r b ∗ , S ∗ ) (w^*, D^*_{\mathrm{orb}}, S^*) ( w ∗ , D orb ∗ , S ∗ ) identifies the natural learning cell of the architecture-data pair ( σ , ρ ) (σ, ρ) ( σ , ρ ) . A new network learns market conditions and predicts stock performance.
problem Optimizing stock portfolio performance in the US equities market.
method Residual Switching Network combining two ResNets: a switching module and a main module.
result The residual switching network strategy outperformed other models with an average annual Sharpe ratio of 2.22.
Effective theory for Transformer initialization improves model performance.
problem Improving performance of Transformers at initialization.
method Effective-theory analysis of signal propagation in wide and deep Transformers.
result Particular width scalings of initialization and training hyperparameters.
GCNIII combines Wide & Deep for better node classification.
problem Issues with graph convolutional networks in node classification tasks.
method Proposes GCNIII framework integrating Wide & Deep architecture and three techniques.
result Demonstrates improved performance in various node classification tasks.
Time-aware deep learning methods improve spatial downscaling of atmospheric pollutants.
problem Transform coarse satellite data of atmospheric pollutants into high-resolution fields.
method Super-resolution deep residual networks and UNet architectures are extended with a temporal module encoding observation time.
result Temporal modules significantly improve downscaling performance and convergence speed.
Proposes a neural network method to correct residual distortions in coordinate transformations.
problem Nonlinear and spatially dependent distortions in coordinate transformation models.
method Residual-based neural network approach focusing on systematic distortions.
result The method improves accuracy and stability in challenging conditions.
Due to the rapid growth of machine learning tools and specifically deep networks in various computer vision and image processing areas, application of Convolutional Neural Networks for watermarking have recently emerged. In this paper, we propose a deep end-to-end diffusion watermarking framework (ReDMark) which can be…
A new algorithm estimates output ranges for deep neural networks efficiently.
problem Estimating output ranges in deep neural networks with complex non-linearities.
method Integrates Simulated Annealing tailored for constrained domains and global optima.
result Guaranteed convergence and robust performance across various DNN architectures.
Paper proposes continuous residual layers for graph neural networks.
problem Low-pass filtering effect in GCN-based models.
method Integrates Ordinary Differential Equations (ODE) to produce outputs of continuous residual layers.
result Continuous residual layers achieve better results than non-residual modules in multiple layers.
Enhanced image denoising with MWRDCNN using residual dense blocks.
problem Image denoising with improved performance and robustness.
method Multi-wavelet residual dense convolutional neural network (MWRDCNN) with residual dense blocks (RDBs).
result Significantly improved performance in image denoising compared to existing techniques.
We conduct mathematical analysis on the effect of batch normalization (BN) on gradient backpropogation in residual network training, which is believed to play a critical role in addressing the gradient vanishing/explosion problem, in this work. By analyzing the mean and variance behavior of the input and the gradient i…
Generalization bounds derived for neural ODEs and deep residual networks.
problem Understanding the generalization capability of neural ODEs and deep residual networks.
method Lipschitz-based argument and analogy with deep residual networks.
result A generalization bound involving the magnitude of weight matrix differences.
Gradient descent converges globally in deep linear residual networks with ZAS initialization.
problem Optimizing deep linear residual networks for convergence.
method Zero-asymmetric (ZAS) initialization for gradient descent.
result Gradient descent converges to an ε-optimal point in O(L^3 log(1/ε)) iterations.
SRFRN accelerates image super-resolution using shallow residual units.
problem High computational complexity and time in deep learning image super-resolution.
method SRFRN uses a bicubic interpolated low-resolution image and residual representative units (RFR) for faster and more efficient high-resolution image reconstruction.
result SRFRN achieves superior performance and faster execution time compared to existing methods.
Residual Continual Learning prevents forgetting in sequential tasks.
problem Preventing catastrophic forgetting in sequential learning of multiple tasks.
method ResCL reparameterizes network parameters by combining original and fine-tuned networks, keeping network size constant.
result ResCL achieves state-of-the-art performance in various continual learning scenarios.
Residual networks with block width max(d_x, d_y) approximate all functions.
problem Achieving universal approximation with residual networks.
method Established bounds on block width for different activation functions.
result Minimum block width for universal approximation is max(d_x, d_y) with inner width 1.
Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.
problem The challenge of hyperparameter tuning in deep learning, especially for large models.
method Combining μ μ μ P parameterization with residual networks having a residual branch scale of 1 / e x t d e p t h 1/\sqrt{ ext{depth}} 1/ e x t d e pt h . result Optimal hyperparameters transfer across width and depth in residual networks trained with this parameterization.
Deep residual networks can approximate any continuous function using control theory.
problem Universal approximation capabilities of deep residual neural networks.
method Relating residual networks to control systems and using Lie algebraic techniques.
result Deep residual networks with adequately deep layers can approximate any continuous function on a compact set.
Residual Networks are shown to be equivalent to boosting feature representation.
problem Improving feature representation in deep learning models.
method Proved ResNet's equivalence to Online Gradient Boosting and proposed decision tree residual modules.
result ResNet can achieve Online Gradient Boosting regret bounds through architectural changes.
GenInSAR uses CNNs to filter InSAR phase and estimate coherence without supervision.
problem Improving accuracy in InSAR phase filtering and coherence estimation.
method Unsupervised CNN-based generative model for joint phase filtering and coherence estimation.
result GenInSAR outperforms five related methods in residue reduction and coherence estimation.
Residual connections significantly boost the performance of deep neural networks. However, there are few theoretical results that address the influence of residuals on the hypothesis complexity and the generalization ability of deep neural networks. This paper studies the influence of residual connections on the hypoth…