New models explain residual and dilated dense neural networks using sparse coding.
problem Lack of theoretical understanding of residual and dilated dense neural networks.
method Proposed Res-CSC and MSD-CSC models, derived mathematical relationships, implemented ISTA.
result Mathematical understanding of residual and dilated dense neural networks.
We introduce a new deep convolutional neural network, CrescendoNet, by stacking simple building blocks without residual connections. Each Crescendo block contains independent convolution paths with increased depths. The numbers of convolution layers and parameters are only increased linearly in Crescendo blocks. In exp…
Review of neural network expressivity and architectures.
problem Understanding neural network expressivity across different architectures.
method Comprehensive overview of approximation results for various neural network types.
result Deep neural networks offer advantages over shallow ones for specific function classes.
Enhanced image denoising with MWRDCNN using residual dense blocks.
problem Image denoising with improved performance and robustness.
method Multi-wavelet residual dense convolutional neural network (MWRDCNN) with residual dense blocks (RDBs).
result Significantly improved performance in image denoising compared to existing techniques.
NESTA accelerates neural networks by compressing Hamming weights.
problem Efficiently computing convolution layers in deep neural networks.
method NESTA reformats convolutions into 3 i m e s 3 3 imes 3 3 im es 3 batches and uses Hamming Weight Compressors to process each batch, approximating partial sums and adding residuals. result Significantly speeds up convolution computations with reduced energy consumption.
DeepcomplexMRI uses deep residual networks for faster MRI imaging.
problem Accelerating parallel MR imaging with high accuracy.
method Deep complex convolutional neural network with residual connections and k-space consistency.
result The method can accurately reconstruct multi-channel MRI images.
BiGraphNet generalizes graph neural networks for more efficient operations.
problem Fragmented graph neural network architectures hinder optimization.
method Explicitly separates input and output nodes, enabling new efficient operations.
result BiGraphNet accelerates and scales computations in hierarchical networks.
Efficient Winograd convolution for INT8 networks using RNS.
problem Difficulty in applying Winograd algorithm to low-precision quantized networks.
method Extends Winograd algorithm to Residue Number System (RNS) for efficient INT8 convolution.
result Arithmetic complexity reduction up to 7.03x with performance improvement up to 2.30x-4.69x.
ResGCN detects anomalies in attributed networks by capturing sparsity and nonlinearity.
problem Detecting anomalous nodes in attributed networks.
method Attention-based deep residual modeling using Graph Convolutional Networks.
result ResGCN effectively detects anomalies in attributed networks.
Deep, wide ConvResNets can approximate functions and their smoothness.
problem Function approximation and smoothness in deep networks.
method Analyzing ConvResNets, proving their ability to approximate functions and their smoothness.
result Large ConvResNets can approximate functions and exhibit sufficient first-order smoothness.
LeanResNet reduces CNN complexity without sacrificing accuracy.
problem High computational cost in CNNs, especially in ResNets.
method Introduces lean convolution operators that reduce parameters and complexity.
result LeanResNet achieves comparable results to other reduced architectures with fewer parameters.
Residual neural networks don't help overcome sampling complexity issues.
problem Learning invertible residual neural networks from samples is hard due to the curse of dimensionality.
method Investigated invertible residual neural networks and their sampling complexity.
result Invertible residual neural networks still suffer from the curse of dimensionality in sampling complexity.
Predicting diagnoses from Electronic Health Records (EHRs) is an important medical application of multi-label learning. We propose a convolutional residual model for multi-label classification from doctor notes in EHR data. A given patient may have multiple diagnoses, and therefore multi-label learning is required. We …
Model based iterative reconstruction (MBIR) algorithms for low-dose X-ray CT are computationally expensive. To address this problem, we recently proposed a deep convolutional neural network (CNN) for low-dose X-ray CT and won the second place in 2016 AAPM Low-Dose CT Grand Challenge. However, some of the texture were n…
Model estimates corporate credibility using NLP and neural networks.
problem Estimating corporate credibility in Chinese listed companies.
method Latent Dirichlet Allocation + Residual Convolutional Neural Network.
result Model ranks companies based on transparency.
Gradient descent efficiently finds global minima in deep neural networks.
problem Training deep neural networks efficiently and reliably.
method Gradient descent, leveraging the stability of the Gram matrix induced by the network architecture.
result Gradient descent achieves zero training loss in polynomial time for deep over-parameterized neural networks with residual connections.
Due to the success of residual networks (resnets) and related architectures, shortcut connections have quickly become standard tools for building convolutional neural networks. The explanations in the literature for the apparent effectiveness of shortcuts are varied and often contradictory. We hypothesize that shortcut…
SRFRN accelerates image super-resolution using shallow residual units.
problem High computational complexity and time in deep learning image super-resolution.
method SRFRN uses a bicubic interpolated low-resolution image and residual representative units (RFR) for faster and more efficient high-resolution image reconstruction.
result SRFRN achieves superior performance and faster execution time compared to existing methods.
Paper analyzes the free energy of CNNs with skip connections in Bayesian learning.
problem Dependency of CNNs with skip connections on the number of parameters.
method Examines the Bayesian free energy of CNNs with and without skip connections.
result The upper bound of free energy of Bayesian CNN with skip connections does not depend on overparametrization.
ResNet with Focal Loss improves speech emotion recognition.
problem Speech emotion recognition using plain text features is insufficient.
method Residual Convolutional Neural Network (ResNet) trained with Focal Loss.
result Focal Loss enhances model's focus on hard examples.
This paper extends geometric study of neural networks to non-differentiable layers and random walks.
problem Understanding the geometric properties of neural networks, especially those with non-differentiable activation functions.
method Singular Riemannian geometry approach to convolutional, residual, and recursive neural networks.
result Illustrated geometric findings with numerical experiments on image classification and thermodynamic problems.
Scattering GCN improves graph neural networks by filtering oversmoothing.
problem Oversmoothing in GCNs limits their ability to distinguish graph nodes.
method Augmenting GCNs with geometric scattering transforms and residual convolutions.
result Scattering GCN outperforms GAT in semi-supervised node classification.
Model learns individual preferences for photo aesthetics.
problem Lack of personalized aesthetics models in photography.
method Residual learning approach to adapt to individual preferences.
result Surpasses state-of-the-art methods in predicting aesthetic value.
RDL-Net improves speech enhancement with fewer parameters and better performance.
problem Improving speech enhancement with fewer parameters and better performance.
method Proposes RDL-Net, a CNN combining residual and dense aggregations without over-allocating parameters.
result RDL-Net achieves higher speech enhancement performance with fewer parameters and lower computational requirements.
An emerging design principle in deep learning is that each layer of a deep artificial neural network should be able to easily express the identity transformation. This idea not only motivated various normalization techniques, such as \emph{batch normalization}, but was also key to the immense success of \emph{residual …
Vision Transformers show different internal representations compared to CNNs.
problem Understanding how Vision Transformers solve image classification tasks.
method Comparative analysis of ViT and CNN architectures on image classification benchmarks.
result ViT has more uniform representations across all layers, while CNNs have more varied representations.
GT-DDP optimizer trains residual networks using game theory.
problem Applying DDP to residual networks.
method Game-theoretic DDP optimizer for residual networks.
result Improved training convergence and variance reduction.
Unified ODE model explains residual and non-residual networks.
problem Unclear relationship between residual and non-residual networks.
method Introducing a damping term in an ODE model to interpolate between ResNet and CNN architectures.
result Unified framework for understanding residual and non-residual networks.
A new deep learning model improves robustness and efficiency in predicting continuous variables.
problem Limited applicability of deep learning in domains with small sample sizes.
method Autoencoder-based residual deep network with shortcut connections.
result Achieves cutting-edge accuracy and efficiency in multiple datasets.
Convolutional neural network for probabilistic time series forecasting.
problem Forecasting multiple related time series with complex patterns.
method Temporal convolutional neural network with stacked residual blocks and dilated causal convolution.
result Outperforms state-of-the-art methods in accuracy and efficiency.
Residual Continual Learning prevents forgetting in sequential tasks.
problem Preventing catastrophic forgetting in sequential learning of multiple tasks.
method ResCL reparameterizes network parameters by combining original and fine-tuned networks, keeping network size constant.
result ResCL achieves state-of-the-art performance in various continual learning scenarios.
A novel method designs multi-block neural networks using Q-learning.
problem Designing optimal neural network architectures efficiently.
method Reinforcement learning (Q-learning) to sequentially pick different types of blocks.
result Effective in creating multi-block neural networks with comparable or better performance.
Deep neural nets classify sleep stages from raw PSG data with high accuracy.
problem Automatically classifying sleep stages from raw polysomnogram signals.
method Deep residual neural networks trained on 50 convolutional layers.
result Best model achieved 84.1% accuracy and 0.746 Cohen's kappa.
New multigrid approach reduces CNN parameters by focusing on structured convolutions.
problem Redundancy in standard CNNs leads to high parameter count.
method Replace standard convolutions with structured multilevel convolutions.
result Linearly proportional number of parameters to network width, no loss in accuracy.
Paper proposes automated depression screening using deep CNNs.
problem Subjective and inconsistent depression diagnosis.
method Convolutional Neural Networks (CNN) and multipart interactive training.
result ResNet architectures achieved 77% accuracy in depression detection.
Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.
problem The challenge of hyperparameter tuning in deep learning, especially for large models.
method Combining μ μ μ P parameterization with residual networks having a residual branch scale of 1 / e x t d e p t h 1/\sqrt{ ext{depth}} 1/ e x t d e pt h . result Optimal hyperparameters transfer across width and depth in residual networks trained with this parameterization.
A new method compresses deep neural networks by predicting and quantizing weights between layers.
problem Resource constraints in deep neural networks.
method Inter-Layer Weight Prediction (ILWP) and quantization based on Smoothly Varying Weight Hypothesis (SVWH).
result The method achieves higher weight compression rates at the same accuracy level.
Improved anomaly detection in time series data using kervolutional neural networks.
problem Anomaly detection in time series data.
method Mixed kervolutional and convolutional layers in a temporal auto-encoder.
result The mixed model detects anomalies more sensitively in time series data.
Proposes a deep diffusion watermarking framework for secure image processing.
problem Secure watermarking of deep neural networks in image processing.
method End-to-end training of two FCNNs with residual structure for embedding and extraction.
result Framework improves security and robustness with customizable trade-offs.
ACNN improves medical image segmentation with full resolution and higher IoUs.
problem Reduced spatial resolution in current DCNNs hinders medical image segmentation.
method Proposes ACNN using atrous convolution, cascaded atrous II-blocks, residual learning, and Instance Normalization.
result ACNN achieves higher IoUs than U-Net and Deeplabv3+ with fewer parameters.
ConvResNets approximate Besov functions and classify on low-dimensional manifolds.
problem Lack of statistical theories for deep learning on high-dimensional data.
method Exploits low-dimensional geometric structures of real-world data sets using ConvResNets.
result ConvResNets can approximate Besov functions and learn classifiers with optimal excess risk.
A convolutional sequence to sequence non-intrusive load monitoring model is proposed in this paper. Gated linear unit convolutional layers are used to extract information from the sequences of aggregate electricity consumption. Residual blocks are also introduced to refine the output of the neural network. The partiall…
GResNet tackles suspended animation in deep GNNs by adding extensive connections.
problem Performance degradation of deep GNNs, especially spectral-based models.
method Introducing GResNet framework with extensively connected highways.
result Avoids dramatic changes to node representations between layers, enhancing learnability.
Efficient facial feature learning with shared representations reduces redundancy and improves accuracy.
problem Redundancy and high computational load in training deep ensemble models.
method Wide Ensemble-based Convolutional Neural Networks (ESRs) with varying branching levels.
result ESRs reduce residual generalization error and outperform state-of-the-art methods on facial expression recognition.
A CNN method removes atmospheric distortion from video sequences.
problem Mitigating atmospheric distortion in video sequences.
method End-to-end supervised convolutional neural network (CNN) with residual learning.
result The method outperforms existing techniques in terms of image quality and speed.
Random forest and deep neural network are two schools of effective classification methods in machine learning. While the random forest is robust irrespective of the data domain, the deep neural network has advantages in handling high dimensional data. In view that a differentiable neural decision forest can be added to…
Residual Networks are shown to be equivalent to boosting feature representation.
problem Improving feature representation in deep learning models.
method Proved ResNet's equivalence to Online Gradient Boosting and proposed decision tree residual modules.
result ResNet can achieve Online Gradient Boosting regret bounds through architectural changes.
PET image reconstruction is challenging due to the ill-poseness of the inverse problem and limited number of detected photons. Recently deep neural networks have been widely and successfully used in computer vision tasks and attracted growing interests in medical imaging. In this work, we trained a deep residual convol…