The study quantifies deep learning generalization error using data distribution and network smoothness.
problem Understanding the generalization error in deep learning models.
method Introducing cover complexity (CC) to measure data difficulty and using the inverse of the modulus of continuity to quantify neural network smoothness. A bound for expected accuracy/error is derived considering both CC and neural network smoothness.
result The expected error of trained neural networks scales with the square root of the number of classes and has a linear relationship with respect to the cover complexity.
Deep neural networks with piecewise-polynomial activations can approximate smooth functions and their derivatives.
problem Approximating smooth functions and their derivatives with neural networks.
method Derives the depth, width, and sparsity required for approximation in Hölder norms.
result Deep neural networks with bounded weights can approximate Hölder smooth functions and their derivatives.
Characterizes neural kernel and NNGP for various activations.
problem Understanding neural kernels and NNGP for non-RELU activations.
method Characterization of RKHS for various activation functions.
result Broad class of non-infinitely smooth activations generate equivalent RKHSs at different depths.
Smooth activations enable optimal error rates in neural networks for Sobolev function classes.
problem Achieving optimal approximation and estimation error rates for neural networks in Sobolev function classes.
method Study of neural networks with smooth activations, proving optimal rates via approximation and statistical properties.
result Constant-depth networks with smooth activations achieve optimal rates of approximation and estimation, demonstrating smoothness adaptivity.
Graph neural networks over-smooth when layers increase, reducing discriminative power.
problem Over-smoothing in graph neural networks reduces model performance as the number of layers increases.
method Analyzed over-smoothing in general graph neural network architecture using Dirichlet energy.
result The Dirichlet energy of embeddings converges to zero, leading to loss of discriminative power.
Deep neural networks with various activation functions can approximate Hölder smooth functions.
problem Expressivity of deep neural networks with general activation functions.
method Investigates approximation ability of deep neural networks with a broad class of activation functions, including Hölder smooth functions.
result Derives the required depth, width, and sparsity of deep neural networks to approximate Hölder smooth functions.
Spatial smoothing improves BNNs' accuracy, uncertainty, and robustness without increasing computational cost.
problem Large ensembles in BNNs increase computational cost and reduce performance.
method Spatial smoothing adds blur layers to convolutional neural networks to ensemble neighboring feature map points.
result Spatial smoothing improves BNNs' performance with fewer ensembles and enhances robustness.
Rational neural networks approximate functions more efficiently with less depth.
problem Choosing optimal nonlinear activation functions in neural networks.
method Rational activation functions with optimal bounds and efficiency proofs.
result Rational neural networks approximate smooth functions more efficiently than ReLU networks with exponentially smaller depth.
Deep neural networks can learn smooth functions without parameters.
problem Learning smooth functions from shallow ReLU neural networks.
method Using over-parameterized shallow ReLU neural networks with norm constraints.
result Least squares estimators based on shallow neural networks are minimax optimal.
Smooth kernel regularizer improves deep neural networks' performance with less data.
problem Deep neural networks need large datasets for effective learning.
method Proposes a smooth kernel regularizer that encourages spatial correlations in convolution kernel weights, learned from previous experience.
result The smooth kernel regularizer improves visual recognition models over an L2 regularization baseline.
The paper bounds the excess risk of deep neural networks for weakly dependent processes.
problem Learning with weakly dependent data using deep neural networks.
method Approximation of smooth functions by deep neural networks and a bound on excess risk.
result The excess risk bound for deep learning under weak dependence is close to O(n−1/2) for sufficiently smooth functions. KuramotoGNN uses Kuramoto model to prevent over-smoothing in graph neural networks.
problem Over-smoothing in graph neural networks where node features become indistinguishable.
method Integrates Kuramoto model to prevent phase synchronization and instead achieve frequency synchronization.
result KuramotoGNN reduces over-smoothing on various graph deep learning tasks.
Develops wavelet-based neural network approximation theory.
problem Analyzing neural network approximation capabilities over various activation functions.
method Wavelet frame theory on spaces of homogeneous type, sufficient conditions for approximation, error estimates.
result Derives sufficient conditions for neural networks to approximate any functions in a given space, including non-smooth activations.
The paper provides approximation guarantees for neural networks trained with gradient flow.
problem Approximating neural networks trained with gradient flow in continuous L2(Sd−1)-norm. method NTK argument for non-convex second but last layer, under-parametrized regime.
result Gradient flow convergence guarantees for neural networks under Sobolev smoothness assumptions.
New neural network smoothness constraints improve model performance.
problem Improving model sensitivity to input changes for better generalization and robustness.
method Exploring current smoothness constraints and proposing new flexible definitions.
result Current smoothness constraints lack flexibility and understanding of data, tasks, and learning.
Exponentially smoothed RNNs improve industrial forecasting.
problem Complexity and non-stationarity in industrial time series data.
method Exponential smoothed recurrent neural networks (RNNs) for modeling non-linear dynamics.
result Exponentially smoothed RNNs outperform traditional models in multi-step forecasting.
In this paper we show how to augment classical methods for inverse problems with artificial neural networks. The neural network acts as a prior for the coefficient to be estimated from noisy data. Neural networks are global, smooth function approximators and as such they do not require explicit regularization of the er…
PDEs constrain smooth functions in neural networks.
problem Understanding functions computable by neural networks.
method Analyzing smooth hierarchical functions via PDEs.
result Established algebraic PDEs for smooth functions.
New findings show learning deeper neural networks is hard even with Gaussian inputs and non-degenerate weights.
problem The computational complexity of learning neural networks, especially deeper ones.
method Smoothed analysis framework and local pseudorandom generators.
result Learning depth-3 ReLU networks under Gaussian input distribution is hard even if weight matrices are non-degenerate.
Deep neural networks with specific parameter sets can approximate smooth functions efficiently.
problem Approximating smooth functions with deep neural networks.
method Deep neural networks with ReLU activation and specific parameter sets {0,±21,±1,2} are used to approximate Cβ-smooth functions. result The constructed networks can approximate Cβ-smooth functions with parameters {0,±21,±1,2} efficiently, achieving the same convergence rate as sparse networks with parameters in [−1,1]. Improves confidence calibration in neural networks by smoothing labels based on class similarity.
problem Improving confidence calibration in deep neural networks for safety-critical applications.
method Proposes a novel label smoothing technique where label values are based on similarities with the reference class, using different similarity measurements.
result Consistently outperforms state-of-the-art calibration techniques on various datasets and network architectures.
Deep, wide ConvResNets can approximate functions and their smoothness.
problem Function approximation and smoothness in deep networks.
method Analyzing ConvResNets, proving their ability to approximate functions and their smoothness.
result Large ConvResNets can approximate functions and exhibit sufficient first-order smoothness.
The paper shows how random ReLU networks converge to smooth splines.
problem Understanding the behavior of shallow ReLU neural networks with random weights.
method Mathematical analysis of L2-regularized regression and gradient descent.
result Random ReLU networks converge to smooth splines as the number of hidden nodes increases.
A neural network with a single hidden layer can't represent certain multivariable functions.
problem Representing certain multivariable functions with a neural network having only one hidden layer.
method Developed a continuum version of a one-hidden-layer neural network with ReLU activation, and proved constraints on its parameters and second derivative.
result Existence of a smooth binary function that cannot be precisely represented by any such neural network.
This work uses sampling theory to analyze smoothness and error bounds of finite neural networks.
problem Analyzing the function space of finite neural networks and providing error bounds.
method Applying sampling theory to finite neural networks with non-expansive activation functions, considering both deterministic and random sampling.
result Novel error bounds for univariate neural networks under band-limited input assumption, highlighting the advantage of deterministic uniform sampling.
Neural networks can learn relationships that traditional models cannot.
problem Identifying factors that differentiate neural networks from traditional models.
method Proving non-identifiability of neural networks compared to smooth parametric models.
result Neural networks can learn nontrivial relationships that traditional models cannot.
Gradient descent in neural networks maximizes margin.
problem Optimizing neural networks using gradient descent.
method Gradient descent or gradient flow on homogeneous neural networks.
result Normalized margin increases over time if training loss decreases below a threshold.
Optimal rates for shallow ReLU networks in nonparametric regression.
problem Approximating smooth and non-smooth functions with shallow ReLU networks.
method Analysis of shallow ReLUk neural networks, using variation norms and deep learning theory. result Optimal approximation rates for shallow ReLU networks in nonparametric regression.
Novel method for nowcasting implied volatility using neural operators.
problem Dynamic and spatially changing option prices in financial markets.
method Operator Deep Smoothing using graph neural operators.
result Highly accurate implied volatility smoothing on ten years of S&P 500 options data.
New neural network with RePU activation approximates smooth functions and their derivatives.
problem Approximating smooth functions and their derivatives with neural networks.
method Differentiable neural networks with RePU activation functions.
result Improved approximation error bounds for RePU-activated neural networks.
FNNs detect EEG signals without position dependence.
problem Detecting EEG signals without position dependence.
method Shift invariant functional neural networks (FNNs) using FDA methods.
result FNNs outperform FDA benchmarks in EEG classification.
Revisits the connection between neural networks and the Kolmogorov-Arnold theorem.
problem Explains the limitations of using the Kolmogorov-Arnold theorem to explain neural networks with multiple hidden layers.
method Derives modifications of the Kolmogorov-Arnold representation that transfer smoothness properties to the outer function and can be well approximated by ReLU networks.
result Shows that a deep neural network with most layers approximating the interior function is a more natural interpretation of the Kolmogorov-Arnold representation.
Study reveals how neural network smoothness affects their vulnerability to adversarial attacks.
problem Understanding adversarial vulnerability in deep learning networks.
method Analysis of manifold smoothness and generalization capability of deep neural networks trained with local errors.
result High generalization accuracy requires a fast power-law decay of eigen-spectrum of hidden representations.
Cubic spline smoothing improves interpolation between irregularly sampled data.
problem Interpolation discontinuity in recurrent neural networks for irregularly sampled sequences.
method Cubic spline smoothing compensation module trained end-to-end with ODE-RNN.
result Improves interpolation between irregularly sampled data points.
Deep ReLU networks can approximate and learn smooth functions efficiently.
problem Efficiently approximating and learning smooth functions using deep ReLU neural networks.
method Extending recent results to anisotropic and mixed smooth function classes, establishing approximation rates.
result Deep ReLU networks achieve minimax optimal rates up to logarithmic factors for various smooth function classes.
New neural networks converge faster, approximating smooth functions better.
problem Understanding the learnability of neural networks through statistical risk.
method Variation-constrained neural networks with arbitrary width.
result Achieves near-parametric rate n−1/2+δ for smooth functions. ConquerNet smooths quantile regression for deep learning with minimax guarantees.
problem Optimization challenges in quantile regression for deep models.
method ConquerNet uses convolution-smoothed quantile ReLU neural networks.
result ConquerNet provides minimax guarantees and outperforms standard quantile neural networks.
Graph pruning improves neural network performance by addressing squashing and smoothing issues.
problem Over-squashing and over-smoothing in Graph Neural Networks.
method Proposes edge deletions to simultaneously address over-squashing and over-smoothing, optimizing spectral gap.
result Edge deletions improve generalization and distinguishability of nodes of different classes.
New findings suggest Barron space doesn't defy curse of dimensionality for certain types of smoothness.
problem Understanding the curse of dimensionality in neural networks with different smoothness notions.
method Defined ADZ spaces via Mellin transform to encapsulate nonclassical smoothness, compared to classical smoothness.
result Evidence provided that Barron space doesn't defy curse of dimensionality for certain smoothness types.
Label smoothing improves model calibration and generalization but harms distillation.
problem Understanding the effects of label smoothing on model calibration and distillation.
method Empirical evaluation and visualization of network representations.
result Label smoothing improves model calibration but harms knowledge distillation.
New method enhances neural network robustness against adversarial attacks.
problem Enhancing neural network robustness against adversarial attacks.
method Variational framework with per-sample noise level selector.
result Enhanced empirical robustness and certified robustness.
The study proves a quantitative functional CLT for neural networks with smooth activation functions.
problem Understanding the convergence rates of neural networks with different activation functions.
method Functional versions of the Stein-Malliavin approach and a quantitative functional central limit theorem.
result Rates of convergence depend on the smoothness of the activation function, ranging from logarithmic to sqrt(n).
New insights into why neural networks can overfit without interpolating data.
problem Understanding why neural networks can overfit without interpolating data in fixed dimensions.
method Analyzing the smoothness of estimators and their derivatives.
result Benign overfitting is possible with estimators that have large enough derivatives, not just in high dimensions but also in fixed dimensions.
Graph Random Neural Network improves semi-supervised learning on graphs.
problem Over-smoothing, non-robustness, and weak-generalization in GNNs with few labeled nodes.
method Random propagation strategy and consistency regularization.
result Significantly outperforms state-of-the-art GNN baselines on semi-supervised node classification.
Smooth neural TPPs using B-splines for better efficiency and accuracy.
problem Efficiently modeling sequences of events in continuous time with neural networks.
method Directly parametrize the CIF as a non-negative combination of B-spline basis functions, predicting coefficients with a neural network.
result Improved computational efficiency and predictive accuracy compared to existing methods.
New RBF networks can approximate any continuous function.
problem Approximating any continuous function on a compact subset.
method Replacing smoothing factors with shifts in RBF networks and proving approximation under certain conditions.
result RBF networks can approximate any continuous function on any compact subset.
Deep neural networks (DNNs) have demonstrated dominating performance in many fields; since AlexNet, networks used in practice are going wider and deeper. On the theoretical side, a long line of works has been focusing on training neural networks with one hidden layer. The theory of multi-layer networks remains largely …
New algorithm speeds up RNN time series prediction by filtering noise.
problem Predicting smooth trajectories from noisy time series data.
method Analyzed RNN dynamics to propose an efficient noise filtering algorithm.
result Significant speedup in predictive process without accuracy loss.