Proposes EMG mixture model for spectroscopy data.
problem Modeling residuals in spectroscopy data with positive support.
method Exponentially-modified Gaussian mixture (EMG) model with expectation-maximization algorithm.
result EMG mixture outperforms existing models in spectroscopy applications.
This research improves asset life prediction by integrating deep learning with mixture distributions.
problem Predicting residual useful life for assets with multiple failure modes.
method Integrates mixture (log)-location-scale distribution with deep learning.
result Proposed models outperform existing methods in predicting residual useful life.
DeRegiME forecasts with regime structure, improving probabilistic predictions across various time series.
problem Probabilistic forecasting discards residual uncertainty, and distribution shifts are hard to capture.
method DeRegiME uses a sparse variational Gaussian process with a nonstationary regime-mixing kernel to separate latent uncertainty regimes.
result DeRegiME improves NLPD by 20.3% on average across benchmarks, with gains on CRPS and MSE.
In this paper we are interested in the problem of learning an over-complete basis and a methodology such that the reconstruction or inverse problem does not need optimization. We analyze the optimality of the presented approaches, their link to popular already known techniques s.a. Artificial Neural Networks,k-means or…
DRMMs enable flexible conditional sampling for interactive machine learning.
problem Limited flexibility in conditional sampling for deep generative models.
method Proposes Deep Residual Mixture Models (DRMMs) that allow flexible conditional sampling.
result DRMMs enable sampling with arbitrary combinations of conditioning variables and priors.
ResMixNet models learn invariant relational reasoning tasks with fewer parameters.
problem Learning invariant relational reasoning tasks in images with random transformations.
method Introduced Residual Mixture Network (ResMixNet) with a mixture-of-experts architecture.
result ResMixNet models achieve less than 2% test error on MNIST Parity task and less than 1% on colorized Pentomino task.
DM framework improves robustness and efficiency in latent-mixture models.
problem Efficient and robust inference in latent-mixture models.
method Divergence-minimization framework with monotonic convergence and robustness guarantees.
result DM yields consistent and asymptotically normal estimators under correct specification.
The thesis evaluates and compares extreme mixture models in finance and insurance.
problem Estimating tail risk measures in finance and insurance.
method Extreme mixture models and methods, including kernel density estimation and GARCH preprocessing.
result Kernel density estimation-based models do not outperform others in tail risk estimation.
Wide residual networks generalize well with uniform convergence to RNTK as width increases.
problem Understanding the generalization ability of wide residual networks.
method Uniform convergence of residual network kernel to residual neural tangent kernel (RNTK).
result Generalization error converges to kernel regression error with respect to RNTK.
Batch normalization makes deep residual networks train faster.
problem Training deep residual networks with large depths.
method Downscaling the residual branch by a normalizing factor early in training.
result Normalized residual blocks compute functions close to the identity function early in training.
Unified ODE model explains residual and non-residual networks.
problem Unclear relationship between residual and non-residual networks.
method Introducing a damping term in an ODE model to interpolate between ResNet and CNN architectures.
result Unified framework for understanding residual and non-residual networks.
ResGCN detects anomalies in attributed networks by capturing sparsity and nonlinearity.
problem Detecting anomalous nodes in attributed networks.
method Attention-based deep residual modeling using Graph Convolutional Networks.
result ResGCN effectively detects anomalies in attributed networks.
Deep residual networks trained with gradient descent have small generalization gap.
problem Limited theoretical understanding of why residual networks generalize well.
method Analyzing overparameterized deep residual networks trained by gradient descent.
result Demonstrates that residual networks have a small generalization gap between training and test error.
Analysis shows BN prevents gradient vanishing/explosion in residual networks.
problem Gradient vanishing/explosion problem in residual networks.
method Mathematical analysis of BN and residual network training.
result BN confounds gradient variance, preventing vanishing/explosion.
Paper introduces statistical CRT for robust multiple parameter estimation.
problem Ambiguity resolution problem with exponential failure probability.
method Proposes a wrapped Gaussian mixture model and two novel approaches for robust estimation.
result Statistically based scheme achieves stronger robustness, especially in low SNR.
The paper introduces a new class of multivariate mixtures for actuarial applications.
problem Developing a new class of multivariate mixtures for actuarial calculations.
method Proposed a class of multivariate matrix-exponential affine mixtures with matrix-exponential marginals.
result Explicit calculations of actuarial quantities are possible due to the proposed class's properties.
Residual neural networks don't help overcome sampling complexity issues.
problem Learning invertible residual neural networks from samples is hard due to the curse of dimensionality.
method Investigated invertible residual neural networks and their sampling complexity.
result Invertible residual neural networks still suffer from the curse of dimensionality in sampling complexity.
A new network learns market conditions and predicts stock performance.
problem Optimizing stock portfolio performance in the US equities market.
method Residual Switching Network combining two ResNets: a switching module and a main module.
result The residual switching network strategy outperformed other models with an average annual Sharpe ratio of 2.22.
iGNN tackles inverse graph prediction using invertible neural networks.
problem Inverse graph prediction problem in data analysis and machine learning.
method Developed invertible graph neural network (iGNN) to solve inverse prediction problem on graphs.
result iGNN model allows efficient generation from output labels and forward prediction.
Residuals improve deep neural networks without increasing hypothesis complexity.
problem Understanding how residual connections affect hypothesis complexity and generalization.
method Analyzing the covering number of the hypothesis space and deriving a margin-based generalization bound.
result Residual connections do not increase the hypothesis complexity of neural networks.
Proposes a neural network method to correct residual distortions in coordinate transformations.
problem Nonlinear and spatially dependent distortions in coordinate transformation models.
method Residual-based neural network approach focusing on systematic distortions.
result The method improves accuracy and stability in challenging conditions.
Enhanced image denoising with MWRDCNN using residual dense blocks.
problem Image denoising with improved performance and robustness.
method Multi-wavelet residual dense convolutional neural network (MWRDCNN) with residual dense blocks (RDBs).
result Significantly improved performance in image denoising compared to existing techniques.
Paper proposes continuous residual layers for graph neural networks.
problem Low-pass filtering effect in GCN-based models.
method Integrates Ordinary Differential Equations (ODE) to produce outputs of continuous residual layers.
result Continuous residual layers achieve better results than non-residual modules in multiple layers.
Generalization bounds derived for neural ODEs and deep residual networks.
problem Understanding the generalization capability of neural ODEs and deep residual networks.
method Lipschitz-based argument and analogy with deep residual networks.
result A generalization bound involving the magnitude of weight matrix differences.
Gradient descent converges globally in deep linear residual networks with ZAS initialization.
problem Optimizing deep linear residual networks for convergence.
method Zero-asymmetric (ZAS) initialization for gradient descent.
result Gradient descent converges to an ε-optimal point in O(L^3 log(1/ε)) iterations.
Residual networks analyzed using linearization for stability under perturbations.
problem Understanding the behavior of residual networks under small input perturbations.
method Linearization of residual units and network stages, using singular value decomposition for stability analysis.
result Most singular values of residual units are 1, but scaling and weights significantly affect them.
SRFRN accelerates image super-resolution using shallow residual units.
problem High computational complexity and time in deep learning image super-resolution.
method SRFRN uses a bicubic interpolated low-resolution image and residual representative units (RFR) for faster and more efficient high-resolution image reconstruction.
result SRFRN achieves superior performance and faster execution time compared to existing methods.
Deep residual networks implicitly converge to neural ODEs.
problem Link between discrete and continuous deep learning models.
method Establishing implicit regularization for residual networks towards neural ODEs.
result Deep residual networks initialized as discretizations of neural ODEs converge to such ODEs during training.
Residual Continual Learning prevents forgetting in sequential tasks.
problem Preventing catastrophic forgetting in sequential learning of multiple tasks.
method ResCL reparameterizes network parameters by combining original and fine-tuned networks, keeping network size constant.
result ResCL achieves state-of-the-art performance in various continual learning scenarios.
Residual networks with block width max(d_x, d_y) approximate all functions.
problem Achieving universal approximation with residual networks.
method Established bounds on block width for different activation functions.
result Minimum block width for universal approximation is max(d_x, d_y) with inner width 1.
Residual Flows improve flow-based models for density estimation.
problem Density estimation using flow-based models with biased log-density estimates.
method Proposed a Russian roulette estimator for unbiased log-density estimation and used an alternative infinite series for gradient calculation. Improved invertible residual blocks with activation functions avoiding derivative saturation and generalized Lipschitz condition to induced mixed norms.
result Residual Flows achieve state-of-the-art performance on density estimation and outperform coupling block networks in joint generative and discriminative modeling.
Residual networks improve noise stability, leading to better performance.
problem Degradation problem in training Residual Neural Networks.
method Introduced simplified versions of ResNets and PlnNets, showing equivalence in expressive power.
result Residual networks have better noise stability, leading to improved generalization.
Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.
problem The challenge of hyperparameter tuning in deep learning, especially for large models.
method Combining μ μ μ P parameterization with residual networks having a residual branch scale of 1 / e x t d e p t h 1/\sqrt{ ext{depth}} 1/ e x t d e pt h . result Optimal hyperparameters transfer across width and depth in residual networks trained with this parameterization.
Deep residual networks can approximate any continuous function using control theory.
problem Universal approximation capabilities of deep residual neural networks.
method Relating residual networks to control systems and using Lie algebraic techniques.
result Deep residual networks with adequately deep layers can approximate any continuous function on a compact set.
Residual Networks are shown to be equivalent to boosting feature representation.
problem Improving feature representation in deep learning models.
method Proved ResNet's equivalence to Online Gradient Boosting and proposed decision tree residual modules.
result ResNet can achieve Online Gradient Boosting regret bounds through architectural changes.
RDL-Net improves speech enhancement with fewer parameters and better performance.
problem Improving speech enhancement with fewer parameters and better performance.
method Proposes RDL-Net, a CNN combining residual and dense aggregations without over-allocating parameters.
result RDL-Net achieves higher speech enhancement performance with fewer parameters and lower computational requirements.
Neural network residuals isolate and locate unknown faults.
problem Locating unknown faults in industrial systems.
method Neural network-based residuals combining physical insights and machine learning.
result Neural network residuals can isolate and locate unknown faults.
Graph neural networks suffer from oversmoothing, but adding residual connections helps.
problem Oversmoothing in deep graph neural networks where features become indistinguishable.
method Analyzed asymptotic oversmoothing rates with and without residual connections using the multiplicative ergodic theorem.
result Adding residual connections effectively mitigates or prevents oversmoothing.
Study shows how deep residual networks can be analyzed as shallow network ensembles for optimization.
problem Understanding why deep neural networks can be trained to zero loss despite non-convex optimization landscapes.
method Mean-field analysis of deep residual networks, focusing on their continuum limit as a two-layer network.
result Derives the first global convergence result for multilayer neural networks in the mean-field regime.
While training error of most deep neural networks degrades as the depth of the network increases, residual networks appear to be an exception. We show that the main reason for this is the Lyapunov stability of the gradient descent algorithm: for an arbitrarily chosen step size, the equilibria of the gradient descent ar…
The paper uses Gaussian mixture models for Bayesian networks and proposes an optimization algorithm.
problem Modeling nodes in Bayesian networks with complex distributions.
method Gaussian mixture models combined with double iteration algorithm.
result The double iteration algorithm optimizes Gaussian mixture models effectively.
Enhances speech from reverberated and noisy signals using Wide Residual Networks.
problem Improving intelligibility of speech from reverberated and noisy signals.
method Wide Residual Network architecture with residual connections for processing contextually correlated representations.
result Enhanced speech from the proposed method succeeded for both enhancement and speech recognition tasks.
Residual algorithms improve reinforcement learning performance.
problem Distribution mismatch in model-based planning.
method Bidirectional target network technique for residual algorithms.
result Residual reinforcement learning significantly outperforms vanilla methods.
Novel LSTM network predicts pulsar timing residuals with few-shot data.
problem Predicting pulsar timing residuals with limited data.
method Long Short-Term Memory (LSTM) network optimized with model-agnostic meta-learning and particle swarm optimization.
result Robust generalization and accurate predictions across high-frequency test domains with minimal data.
Framework uses deep learning and statistical models to solve PDEs with discontinuous coefficients.
problem Solving PDEs with discontinuous coefficients.
method Two-stage physics-informed deep learning and statistical mixture models.
result Framework achieves adaptability and accurate parameter identification.
MSNet uses high frequency residual learning for efficient multi-scale image classification.
problem Efficient multi-scale image classification for mobile and embedded devices.
method Two network architecture: low resolution for low frequency, high resolution for high frequency residuals.
result MSNet achieves significant accuracy improvements over different base networks.
New method views residual networks as stochastic differential equations.
problem Improving generalization of neural networks.
method Applying modified equations to show residual networks as weak approximations of stochastic differential equations.
result Stochastic training of residual networks can be understood through the lens of optimal control of backward Kolmogorov's equations.
Residual networks' depth is mathematically equivalent to expanding an implicit ensemble size.
problem Understanding why deep residual networks are effective.
method Formal analysis of residual networks as ensembles of shallow models.
result Increasing network depth is equivalent to expanding the size of an implicit ensemble, revealing a hierarchical structure.