Residual networks with block width max(d_x, d_y) approximate all functions.
problem Achieving universal approximation with residual networks.
method Established bounds on block width for different activation functions.
result Minimum block width for universal approximation is max(d_x, d_y) with inner width 1.
Residual flows are shown to approximate MMD well.
problem Lack of theoretical understanding of normalizing flows' expressiveness.
method Proved residual flows are universal approximators in MMD.
result Residual flows can approximate MMD with a bounded number of blocks.
Enhanced image denoising with MWRDCNN using residual dense blocks.
problem Image denoising with improved performance and robustness.
method Multi-wavelet residual dense convolutional neural network (MWRDCNN) with residual dense blocks (RDBs).
result Significantly improved performance in image denoising compared to existing techniques.
AANets balance stability and plasticity in CIL.
problem Stability-plasticity dilemma in class-incremental learning.
method Adaptive Aggregation Networks (AANets) with stable and plastic residual blocks.
result AANets improve performance on CIL benchmarks.
JKO-iFlow uses neural ODEs to improve generative models with reduced memory and training complexity.
problem Efficiently training deep generative models in high dimensions with reduced memory and training complexity.
method JKO scheme inspired neural ODE flow network with adaptive time reparameterization.
result JKO-iFlow achieves competitive performance compared to existing models at reduced computational and memory cost.
GRRT recovers sparse signals without prior sparsity or noise variance knowledge.
problem Recovering sparse signals without prior sparsity or noise variance knowledge.
method Generalized residual ratio thresholding (GRRT) for SOMP and BOMP.
result Finite sample and finite SNR guarantees for exact support recovery.
Paper finds effective building blocks for CNNs, improving model performance and size.
problem Finding optimal deep model architectures and parameters.
method Search framework for discovering effective building blocks for CNNs.
result Discovered models are smaller and perform comparably to state-of-the-art models.
Batch normalization makes deep residual networks train faster.
problem Training deep residual networks with large depths.
method Downscaling the residual branch by a normalizing factor early in training.
result Normalized residual blocks compute functions close to the identity function early in training.
Deep ResNets can have better local minima than linear predictors.
problem Understanding the optimization landscape of deep ResNets compared to linear predictors.
method Analyzing the optimization landscape of ResNets with multiple residual blocks, showing geometric conditions under which ResNets have better local minima.
result Theorem showing that any critical point in the optimization landscape of deep ResNets is either at least as good as the best linear predictor or has a strictly negative eigenvalue in its Hessian.
VLN generates future video frames efficiently with lateral connections.
problem Efficiently generating future video frames.
method Neural encoder-decoder model with lateral connections, including recurrent and feedforward lateral connections.
result VLN achieves competitive results on the Moving MNIST dataset.
ResNets can approximate arbitrary ODEs in space and time.
problem Approximating solutions of arbitrary ODEs in space and time.
method Using ResNets as space-time approximations of ODEs, deriving complexity estimates.
result Deep ReLU ResNets can approximate solutions of arbitrary ODEs in space and time.
Dynamic steerable blocks improve deep networks by learning filter invariances.
problem Pixel-based filters ignore image properties, leading to suboptimal performance.
method Developed frame-based ResNets and Densenets, which are steerable under predefined transformations.
result Dynamic steerable blocks outperform other approaches on contour detection datasets.
Improved ResNets and DenseNets models for better feature reuse.
problem Diminishing feature reuse in ResNets and DenseNets.
method ResNEsts and DenseNEsts are block-based DNN models with improved representation guarantees.
result Wide ResNEsts with bottleneck blocks can guarantee desirable training properties.
Residual Flows improve flow-based models for density estimation.
problem Density estimation using flow-based models with biased log-density estimates.
method Proposed a Russian roulette estimator for unbiased log-density estimation and used an alternative infinite series for gradient calculation. Improved invertible residual blocks with activation functions avoiding derivative saturation and generalized Lipschitz condition to induced mixed norms.
result Residual Flows achieve state-of-the-art performance on density estimation and outperform coupling block networks in joint generative and discriminative modeling.
Two-layer model sparsifies image residuals for CT image reconstruction.
problem Image reconstruction from limited and corrupted data.
method Pre-learning a two-layer sparsifying transform model with block coordinate descent optimization.
result Preliminary experiments show the two-layer model improves CT image reconstruction from low-dose measurements.
Study provides selective inference method for latent block models.
problem Challenges in constructing a test on a block structure selected by clustering algorithms.
method Developed a selective inference method for latent block models using squared residue minimization and simulated annealing.
result Proposed tests effectively handle selective bias in block structures compared to naive tests.
CrescendoNet is a simple deep CNN outperforming others on benchmark datasets.
problem Improving performance of deep neural networks without residual connections.
method Stacking simple blocks with independent convolution paths, increasing depth linearly.
result CrescendoNet with 15 layers and 4.1M parameters outperforms DenseNet-BC with 250 layers and 15.3M parameters.
New deep architecture improves stability and robustness of neural networks.
problem Improving stability and robustness of neural networks.
method Proposes a new deep architecture using implicit residual blocks with fixed point definitions.
result Improved stability and generalization power with fewer hyperparameters.
CN-SBM clusters cancer samples and regions based on copy number variants.
problem Clonal evolution in cancer monitored by noisy copy number variants.
method Probabilistic framework using bipartite categorical block model.
result Improved model fit and clinically relevant subtypes identified.
Paper proposes graph-based separable transforms for video coding.
problem Improving video coding efficiency by better capturing residual block statistics.
method Derives graph-based separable transforms (GBSTs) from line graphs with weights determined by parameters.
result GBSTs achieve about 0.4% average coding gain over existing transforms in VVC.
Analysis shows BN prevents gradient vanishing/explosion in residual networks.
problem Gradient vanishing/explosion problem in residual networks.
method Mathematical analysis of BN and residual network training.
result BN confounds gradient variance, preventing vanishing/explosion.
Accelerating Speculative Diffusions via Block Verification
problem Adapting speculative decoding for continuous diffusion models
method Introducing a novel speculative sampling mechanism for diffusion models
result Improves acceptance rate and speeds up inference
New research shows alternative linear connections can outperform identity shortcuts in deep networks.
problem Explaining the effectiveness of shortcut connections in deep neural networks.
method Used variations of the standard residual block with different types of linear connections to build image classification networks.
result Alternative linear connections can be more effective than identity shortcuts in deep networks.
Sharp stability threshold found for deep residual architectures.
problem Ensuring stable training and inference in deep residual networks.
method Sublinear-growth principle and optimal-control analysis.
result Stable training condition: input-magnitude exponent q ≤ 1.
A novel method designs multi-block neural networks using Q-learning.
problem Designing optimal neural network architectures efficiently.
method Reinforcement learning (Q-learning) to sequentially pick different types of blocks.
result Effective in creating multi-block neural networks with comparable or better performance.
DMPS learns set data by connecting graph and set learning.
problem Lack of relational learning for set data.
method Deep Message Passing on Sets (DMPS) that connects graph and set learning.
result DMPS achieves competitive or superior results on synthetic and real-world datasets.
Convolutional model disaggregates electricity consumption data.
problem Disaggregating aggregate electricity consumption data into individual appliance usage.
method Gated linear unit convolutional layers and residual blocks refine neural network output. Partially overlapped sequences are averaged for final output.
result The proposed model outperforms existing models in disaggregating various appliance usage.
A new network learns market conditions and predicts stock performance.
problem Optimizing stock portfolio performance in the US equities market.
method Residual Switching Network combining two ResNets: a switching module and a main module.
result The residual switching network strategy outperformed other models with an average annual Sharpe ratio of 2.22.
Effective theory for Transformer initialization improves model performance.
problem Improving performance of Transformers at initialization.
method Effective-theory analysis of signal propagation in wide and deep Transformers.
result Particular width scalings of initialization and training hyperparameters.
Deep residual networks improve short-term power load forecasting.
problem Short-term power load forecasting accuracy and generalization.
method Modified deep residual network with two-stage ensemble strategy and Monte Carlo dropout.
result The proposed model provides accurate load forecasting results and high generalization capability.
Unified learning-rate scale for CNNs and ResNets, avoiding depth imbalance.
problem Challenges in choosing an appropriate learning rate for deep networks, especially as depth increases.
method Introduces Arithmetic-Mean μP (AM-μP), constraining network-wide average pre-activation second moment to a constant scale, combined with residual-aware He fan-in initialization. result Demonstrates a −3/2 scaling law for learning rates across depths, enabling zero-shot learning-rate transfer. RSPFs combine multiple SPNs for better density estimation.
problem Creating large SPNs for complex data.
method Random sum-product forests with residual links.
result RSPFs outperform individual SPNs and improve with residual links.
BCD methods provide provable convergence guarantees for deep learning models.
problem Theoretical convergence guarantees for BCD methods in deep learning.
method Established global convergence rate of O(1/k) for DNN training models.
result Global convergence to a critical point at a rate of O(1/k) for most DNN training models.
Study tests how U.S. equity prices align with global asset frequencies using financial variables.
problem Testing whether U.S. equity prices align with global asset frequencies using financial variables.
method Examines SPX and RUT gaps, uses OIS-based funding, volatility, trading-friction, financial-condition variables, and residual information.
result Gains in fit survive broad-dollar neutralization, alternative blocks, PCA, residualization, and nested horizon selection, supporting reduced-form P-Q alignment.
End-to-end image super-resolution using Attention-based DenseNet with residual deconvolution.
problem Challenging task of improving low-resolution images.
method Proposes a novel ADRD model with weighted dense blocks and spatial attention modules.
result Demonstrates promising performance on publicly available datasets.
Extends VAEs to handle complex Bayesian network structures.
problem Handling complex dependency structures in Bayesian networks.
method Extends VAEs with graphical residual flows to model arbitrary dependency structures.
result Demonstrates improved performance on synthetic datasets.
RDL-Net improves speech enhancement with fewer parameters and better performance.
problem Improving speech enhancement with fewer parameters and better performance.
method Proposes RDL-Net, a CNN combining residual and dense aggregations without over-allocating parameters.
result RDL-Net achieves higher speech enhancement performance with fewer parameters and lower computational requirements.
Extended LSTM improves language modeling performance.
problem Improving LSTM for large-scale language modeling.
method Exponential gating, modified memory structure, and residual stacking.
result xLSTM outperforms state-of-the-art models in performance and scaling.
DID solves NMF for large distributed data.
problem Large distributed nonnegative matrix factorization.
method Distributed incremental block coordinate descent.
result Only one communication step per iteration.
ImpFlows generalize normalizing flows by implicitly defining transformations.
problem Creating flexible and tractable probability distributions.
method Implicitly defined invertible transformations using roots of equations.
result ImpFlows can represent functions that ResFlows cannot, with comparable parameters.
ResNets promote smoother interpolations than MLPs, enhancing generalization.
problem Understanding the difference in smoothness between ResNets and MLPs.
method Neural Tangent Kernel (NTK) analysis during gradient descent training.
result ResNet's NTK results in smoother interpolations than MLPs.
Deep neural networks approximate unknown governing equations from data.
problem Approximating unknown governing equations from observational data.
method Residual network (ResNet) and multi-step methods (RT-ResNet, RS-ResNet) for equation recovery.
result Deep neural networks can recover governing equations without time derivative data.
DiffusionBlocks trains neural networks by breaking them into independent blocks, reducing memory usage.
problem Memory bottlenecks in end-to-end neural network training.
method Transforming transformer-based networks into independent trainable blocks via a denoising process.
result Independent block-wise training matches end-to-end training performance while reducing memory requirements.
RFRBoost uses random features to boost deep residual neural networks, improving performance and computational efficiency.
problem Improving performance of deep residual neural networks (RFNNs) while preserving convex optimization benefits.
method Random Feature Representation Boosting (RFRBoost) using boosting theory and random features at each layer.
result RFRBoost significantly outperforms RFNNs and end-to-end trained MLP ResNets in small- to medium-scale tabular datasets.
N-BEATS improves time series forecasting accuracy by 11% over benchmarks.
problem Univariate time series point forecasting problem
method Neural architecture based on backward and forward residual links and fully-connected layers
result State-of-the-art performance on diverse datasets, improving forecast accuracy by 11% over statistical benchmarks
A joint model predicts IT operations and detects anomalies.
problem Predicting IT operations and detecting anomalies in noisy data.
method A joint model combining variational auto-encoder and LSTM, with spectral residual analysis integration.
result The joint model outperforms models trained separately on prediction and anomaly detection tasks.
PSiLON Net uses L1 weight normalization and 1-path-norm regularization for efficient learning and sparsity.
problem Efficient learning and sparsity in neural networks with limited data.
method PSiLON Net employs L1 weight normalization and 1-path-norm regularization to simplify the 1-path-norm and achieve efficient learning and near-sparse parameters. result PSiLON Net achieves reliable optimization and strong performance in the small data regime.
Paper models graph edge dependencies using latent variables for community detection.
problem Graphs' edge dependencies not fully explained by community membership.
method Introduces auxiliary latent variables to model edge dependencies and analyzes conditions for exact recovery.
result Exact recovery possible by semidefinite programming down to maximum likelihood threshold.