Deep convolutional nets are essential for accurate learning on CIFAR-10.
problem Training shallow models to mimic deep convolutional nets on CIFAR-10.
method Used distillation to train shallow feed-forward nets on CIFAR-10, demonstrating the necessity of multiple convolutional layers.
result Accurate models on CIFAR-10 require multiple convolutional layers, even when trained with distillation.
PLN-Nets with two linear layers and parallel LN achieve universal approximation.
problem Limitations of standard neural network architectures in universal approximation.
method Introduced PLN-Nets combining two linear layers with parallel LN.
result PLN-Nets achieve universal approximation, while standard LN has limited power.
Paper transforms deep rectifier networks into shallow ones for analysis.
problem Understanding the complexity of deep neural networks.
method Transformation of deep rectifier networks into shallow ones.
result Shallow networks can represent deep networks with fewer functions.
Deep nets outperform shallow nets in complex feature realization.
problem Realizing complex data features with deep nets.
method Refined covering number estimates and analysis of approximation rates.
result Deep nets can improve performance without additional capacity costs for complex features.
Deep nets improve function approximation and learning in high dimensions.
problem Designing neural networks for rotation-invariant function approximation.
method Developed deep neural networks with multiple hidden layers for radial function approximation.
result Deep nets achieve near-optimal function approximation and learning rates not possible by shallow nets.
Neural nets trained with linear discriminant initialization converge faster and more accurately.
problem Training feed-forward neural networks efficiently and accurately.
method Initialize first layer weights with linear discriminants.
result Asymptotic higher accuracy and faster convergence.
New bound on neural nets complexity for approximating functions.
problem Approximating continuous functions with shallow neural networks.
method Inspired by Stone-Weierstrass theorem, constructive proof.
result General upper bound on neuron count for accuracy.
Deeper neural networks can better approximate certain natural functions than shallower ones.
problem Approximating natural functions with neural networks.
method Depth-based separation results for feed-forward neural networks.
result Deeper networks can better approximate certain types of functions than shallower ones.
Paper tackles invariance of demodulation in shallow water acoustic communications.
problem Frequency-selective signal distortion (Doppler effect) in shallow water environments.
method Developed ML-based demodulation methods using DBN-NN and DBN-CNN.
result Demonstrated invariance of the proposed method to Doppler effect with 2dB error margin.
The study examines how shallow neural nets converge to training samples or manifold points during diffusion.
problem Understanding when and how shallow neural nets converge to training samples or manifold points during diffusion.
method Analysis of shallow ReLU neural network denoisers trained with minimal ℓ2 norm, comparing score flow and diffusion flow. result Probability flow converges to training points, sums of training points, or manifold points, depending on the diffusion time scheduler.
Proposes a learning rate method for shallow nets based on gradient Lipschitz constant.
problem Finding optimal learning rates for shallow neural networks.
method Associates learning rate with gradient Lipschitz constant and proposes a search algorithm.
result The proposed method significantly outperforms existing tuning methods.
AR-Net models time-series with interpretable coefficients and scalability.
problem Modeling time-series with long-range dependencies and interpretability.
method Feed-forward neural network approach to AR-process dynamics.
result AR-Net learns identical AR-coefficients as Classic-AR and scales to long-range dependencies.
Paper learns an optimization algorithm for neural nets.
problem High-dimensional stochastic optimization problems in neural nets.
method Reinforcement learning framework to learn optimization algorithms.
result Learned optimization algorithm outperforms known algorithms.
Feed-forward nets fail to learn equality relations, but adding DR units helps.
problem Feed-forward neural networks struggle to learn equality relations reliably.
method Introduced differential rectifier (DR) units to create an inductive bias.
result DR units enable feed-forward nets to learn equality relations reliably.
New method uses outer product manifolds to simplify neural networks.
problem Exponential inefficiency of hierarchical neural networks.
method Reparametrization invariant Riemannian metrics and tangent subspace computation.
result Significant improvement in network performance after early training.
Physics-informed neural networks solve physics problems using neural nets.
problem Discovering nonlinear PDEs from data.
method Two classes of algorithms: continuous time and discrete time models.
result Demonstrated effectiveness on various physics problems.
Shallow neural nets classify objects perfectly if their distribution is linearly separable.
problem Designing efficient neural networks for classification.
method Constructed shallow sigmoid-type neural networks.
result Achieves 100% accuracy for datasets following a linear separability condition.
Paper identifies neural network weights from few samples using differentiation and tensor products.
problem Identifying weights of shallow neural networks from limited data.
method Uses second-order differentiation and tensor product decomposition to identify weights from a small number of samples.
result Proves successful identification of weight vectors close to orthonormal and constructsively reduces generality.
New method uses neural networks to forecast spatial-temporal data.
problem Probabilistic forecasting of spatio-temporal data with causal structure.
method MMAF-guided learning with ensemble of stochastic feed-forward neural networks.
result Forecasting remains calibrated across multiple time horizons.
Estimates neural network error approximating compact sets.
problem Approximating compact subsets from Banach spaces with neural networks.
method Estimates error rates for neural networks of varying width and depth.
result Depth is crucial for better approximation rates, width alone does not improve.
Minimal width din+1 allows ReLU nets to approximate any continuous function of din variables.
problem Approximating continuous functions using ReLU nets with minimal width.
method Analyzing the expressive power of depth in neural nets with ReLU activations.
result Minimal width din+1 is necessary and sufficient for ReLU nets to approximate any continuous function of din variables. Theory explains how deep nets learn features from data.
problem Understanding how deep neural networks learn features from data.
method Developed a noise-nonlinearity phase diagram and a mechanical theory.
result Links feature learning across layers to generalization.
New method uses MMAF-guided learning for spatio-temporal probabilistic forecasts.
problem Probabilistic forecasting of spatio-temporal data with causal structure.
method Generalized Bayesian methodology, MMAF-guided learning, ensemble of stochastic feed-forward neural networks.
result Forecast performance comparable to, and sometimes better than, deep learning architectures.
Paper uses neural networks to analyze oil price impact on Iranian stock and industry indices.
problem Impact of oil price volatility on Tehran stock and industry indices.
method Feed-forward neural networks analysis of two periods: sanctions and post-sanctions.
result Neural networks predict stock and industry indices well, showing significant oil price volatility impact.
New bound for neural nets on non-iid data.
problem Generalization of deep nets for dependent data.
method Establishes a generalization bound for feed-forward neural networks on φ-mixing data. result Proves neural nets can generalize well on non-iid data.
Neural nets solve braid untangling up to length 20.
problem Untangling braids in knot theory and group theory.
method Feed-forward neural networks in reinforcement learning.
result Trained neural networks to untangle braids in minimal moves.
This work explores the relation between depth and expressivity in neural networks.
problem Understanding the power of depth in neural networks and its relation to gradient-based optimization.
method Depth separation argument for distributions with fractal structure, proving that deep networks can express fine details efficiently but shallow ones cannot.
result The success of learning deep networks depends on whether the distribution can be well approximated by shallower networks.
Gradient descent struggles to learn equivariant neural networks, even with symmetries.
problem Learning equivariant neural networks via gradient descent is hard.
method Lower bounds for various equivariant neural network classes.
result Gradient descent struggles to learn equivariant neural networks, even with symmetries.
New stability bounds for GD in overparameterised shallow nets without NTK assumptions.
problem Generalisation and excess risk bounds for shallow neural networks.
method Oracle inequalities and stability analysis of GD without kernelisation.
result Oracle type bounds reveal GD's generalisation is controlled by an interpolating network with shortest GD path.
GD-trained shallow ReLU nets learn Lipschitz functions with noise.
problem Learning Lipschitz functions with additive noise in overparameterized neural networks.
method Gradient Descent (GD) with early stopping, focusing on the Neural Tangent Kernel (NTK).
result Early-stopped GD achieves minimax optimal rates for learning Lipschitz functions.
GD with early stopping trains shallow neural nets for nonparametric regression robustly.
problem Learning Lipschitz regression functions with noisy labels.
method Overparameterized shallow neural networks trained by GD with early stopping.
result Optimal rates of convergence for nonparametric regression.
RDL-Net improves speech enhancement with fewer parameters and better performance.
problem Improving speech enhancement with fewer parameters and better performance.
method Proposes RDL-Net, a CNN combining residual and dense aggregations without over-allocating parameters.
result RDL-Net achieves higher speech enhancement performance with fewer parameters and lower computational requirements.
A method to simplify deep neural networks for specific tasks.
problem Reducing deep neural networks to a smaller size while maintaining functionality.
method Advanced Supervised Principal Component Analysis-based shallowing algorithm.
result The method can reduce network depth without significant performance loss.
PDE-Net learns PDEs from data using neural networks.
problem Learning PDEs from complex system dynamics.
method Proposes PDE-Net, a feed-forward deep network to learn differential operators and nonlinear responses.
result PDE-Net can accurately predict dynamics and uncover hidden PDE models.
Survey of machine learning methods for time series forecasting.
problem Improving accuracy of time series predictions.
method Linear and nonlinear machine learning models, including neural networks and ensemble methods.
result Demonstrates superior predictive ability of certain machine learning models.
pystacked combines machine learning models for improved predictions.
problem Improving machine learning model performance through stacking.
method Stacked generalization using Python's scikit-learn with various base learners.
result Enhanced predictive models through combining multiple machine learning algorithms.
Survival regression method improves log-likelihood scores.
problem Improper scoring rules in survival regression models.
method SurvivalMonotonic-net (SuMo-net) with monotonic neural networks.
result SuMo-net achieves state-of-the-art log-likelihood scores.
Sample variance decay is shown in deep ReLU networks, impacting training dynamics.
problem Sample variance decay in deep ReLU networks during training.
method Decomposed total variance into sample variance and network-averaged sum of sample mean and variance.
result Sample variance decays in later layers of deep ReLU networks, impacting training dynamics.
LPOM approximates neural nets with proximal operators for efficient training.
problem Training feed-forward neural networks efficiently and avoiding gradient issues.
method Rewriting activation functions as proximal operators, using block coordinate descent.
result LPOM converges and performs well on MNIST and CIFAR-10.
Deep Bayesian neural nets can use simpler weight approximations without sacrificing performance.
problem The need for complex weight posterior approximations in deep Bayesian neural networks.
method Theoretical and empirical analysis of mean-field variational inference in deep networks.
result Mean-field variational weight posteriors in deep networks can induce similar function-space distributions as complex approximations in shallower networks.
A method to reduce latency in DNN-based decision making by gating computations.
problem Resource constraints in test-time deployment of deep neural networks.
method Adaptive gating function that recognizes states for shallow policy sufficiency.
result Up to 5X speed-up in average latency with minimal performance loss.
Gradient descent converges to global optima in shallow overparameterized networks.
problem Understanding the level of overparameterization required for gradient descent to converge to a global optimum in shallow neural networks.
method Stochastic gradient descent starting from random initialization, focusing on shallow neural nets and smooth activations.
result Gradient descent converges geometrically to a nearby global optima as soon as the square-root of the number of parameters exceeds the size of the training data.
DS2CF-Net learns hierarchical representations with deep coupled factorization and enriched prior.
problem Learning deep hierarchical representations from data.
method Dual-constrained Deep Semi-Supervised Coupled Factorization Network (DS2CF-Net) with enriched prior.
result DS2CF-Net achieves state-of-the-art performance in representation learning and clustering.
Space2Vec learns multi-scale spatial representations from grid cell insights.
problem Encoding spatial features with varying scales from GIS data.
method Proposes Space2Vec, a multi-scale representation learning model using grid cell insights.
result Space2Vec outperforms baselines in predicting POI types and image classification with geo-locations.
A hybrid model combines physics and machine learning to predict unknown processes in numerical models.
problem Unknown or poorly represented processes in numerical models of the Earth System.
method Combining a physical model with a neural-net trained on observations.
result The hybrid model accurately predicts unknown processes with high correlation (close to 1).
Neural Additive Models combine neural nets with interpretable models for high stakes tasks.
problem Inability to understand how neural networks make decisions.
method Combines neural nets with generalized additive models to create Neural Additive Models (NAMs).
result NAMs are more accurate than intelligible models and as accurate as state-of-the-art generalized additive models.
A neural network, IHT-Net, improves DOA estimation with sparse arrays.
problem Single-snapshot DOA estimation with sparse arrays in dynamic settings.
method IHT-inspired neural network with recurrent neural network and autoencoders.
result IHT-Net achieves faster convergence and higher accuracy in DOA estimation.
DeepCA combines neural networks with component analysis for improved performance.
problem Limited capacity of shallow component analysis in deep learning.
method Deep Component Analysis (DeepCA) with ADNNs for inference.
result Improved performance on various tasks, including depth prediction.