LSNN improves CNN by smoothing local receptive fields.
problem Limited capturing of local receptive fields in CNN.
method LSNN represents kernel and smoother to capture local fields' importance and relations.
result LSNN outperforms CNN and locally connected layer on MNIST variants.
New algorithm trains neural networks faster than SGD.
problem Training neural networks to local minima efficiently.
method Stochastic algorithm with O ( ε − 3.25 ) O(\varepsilon^{-3.25}) O ( ε − 3.25 ) backpropagations. result Trains to ε \varepsilon ε -approximate local minima faster than SGD. Study reveals how neural network smoothness affects their vulnerability to adversarial attacks.
problem Understanding adversarial vulnerability in deep learning networks.
method Analysis of manifold smoothness and generalization capability of deep neural networks trained with local errors.
result High generalization accuracy requires a fast power-law decay of eigen-spectrum of hidden representations.
Piecewise linear activations create many spurious local minima in neural networks.
problem Understanding the loss surface of neural networks with piecewise linear activations.
method Proved the existence of infinite spurious local minima and partitioned the loss surface into smooth cells.
result Piecewise linear activations create many spurious local minima that are invariant under a continuous path.
Neural networks for binary classification have zero training error at all local minima under certain conditions.
problem Understanding the loss surface of neural networks for binary classification.
method Analyzing single-layered neural networks with smooth hinge loss function, providing conditions for zero training error at all local minima.
result Zero training error at all local minima is achieved under specific conditions (strict convexity of neurons and smooth hinge loss).
New findings show learning deeper neural networks is hard even with Gaussian inputs and non-degenerate weights.
problem The computational complexity of learning neural networks, especially deeper ones.
method Smoothed analysis framework and local pseudorandom generators.
result Learning depth-3 ReLU networks under Gaussian input distribution is hard even if weight matrices are non-degenerate.
A new method boosts graph neural networks by preventing over-smoothing and over-squashing.
problem Graph Neural Networks struggle with long-range signals and over-smoothing/over-squashing.
method Proposes PowerEmbed, a layer-wise normalization technique inspired by spectral graph embedding.
result PowerEmbed prevents over-smoothing and avoids over-squashing, improving performance on heterophilous graphs.
New theory for local parameterization of deep ReLU networks.
problem Determining local parameters of deep ReLU neural networks.
method Introducing local lifting operators and charts of a manifold, deriving necessary and sufficient conditions for local identifiability.
result Sharp and testable conditions for local identifiability of deep ReLU networks.
Deep neural networks can learn smooth functions without parameters.
problem Learning smooth functions from shallow ReLU neural networks.
method Using over-parameterized shallow ReLU neural networks with norm constraints.
result Least squares estimators based on shallow neural networks are minimax optimal.
Graphs can be smoothed or squashed too, study finds.
problem Graph Neural Networks struggle with over-smoothing and over-squashing issues.
method Unified framework using Ollivier-Ricci curvature to address both issues.
result Over-smoothing and over-squashing linked to positive and negative graph curvature respectively.
Neural networks can learn relationships that traditional models cannot.
problem Identifying factors that differentiate neural networks from traditional models.
method Proving non-identifiability of neural networks compared to smooth parametric models.
result Neural networks can learn nontrivial relationships that traditional models cannot.
Shallow neural networks can represent polynomials efficiently.
problem Representing polynomials using shallow neural networks.
method Using shallow neural networks of width 2 ( R + d ) d 2(R+d)^d 2 ( R + d ) d to represent d d d -variate polynomials of degree R R R . result Derives minimax optimal convergence rate for shallow networks to unknown univariate regression functions.
Adaptive smoothing in fMRI improves brain activity analysis.
problem Optimizing spatial smoothing in fMRI data processing pipelines.
method Integrating adaptive spatial smoothing as a neural network layer.
result Adaptive smoothing enhances brain activity analysis accuracy.
Graph Spectral Regularization makes neural network layers more interpretable.
problem Making neural network layers more interpretable without sacrificing performance.
method Using a graph Laplacian penalty to structure hidden layer activations.
result Encourages smooth activations within hidden layers, leading to better interpretability.
New algorithm calibrates local volatility from option prices using deep neural networks.
problem Calibrating local volatility from market option prices with reduced interpolation and reprice errors.
method Deep self-consistent learning using neural networks to approximate both option prices and local volatility.
result Improved performance in terms of reduced interpolation and reprice errors compared to existing methods.
New method improves calibration of neural networks by targeting robust margins and local smoothness.
problem Poor calibration of neural networks, leading to unreliable confidence estimates.
method Intervene on training procedure by targeting robust margins and local smoothness.
result Improved out-of-sample calibration without sacrificing accuracy.
Over-parametrized networks with quadratic activations can find globally optimal solutions for convex losses.
problem Finding globally optimal solutions in neural networks with quadratic activations.
method Analyzing the landscape properties of loss functions and using Rademacher complexity for generalization.
result Over-parametrization with k ≥ 2 n k \ge \sqrt{2n} k ≥ 2 n enables finding globally optimal solutions for convex losses. We use smoothed analysis techniques to provide guarantees on the training loss of Multilayer Neural Networks (MNNs) at differentiable local minima. Specifically, we examine MNNs with piecewise linear activation functions, quadratic loss and a single output, under mild over-parametrization. We prove that for a MNN with …
Deep neural networks can interpolate any dataset in the overparametrized regime.
problem Interpolating any dataset with deep neural networks in the overparametrized regime.
method Proving universal approximations and interpolating any dataset with deep neural networks, considering specific conditions on activation functions.
result Interpolation of any dataset is possible in the overparametrized regime with deep neural networks.
The curse of dimensionality affects neural network optimization, especially with smooth functions.
problem The curse of dimensionality in neural network optimization.
method Examined through the evolution of the parameter distribution under 2-Wasserstein gradient flow.
result The curse of dimensionality persists in neural network optimization, even with smooth functions.
Paper refines RNN training by analyzing smoothness and attractors.
problem Exploding and vanishing gradients in RNNs.
method Refined concept of exploding gradients using cost function smoothness.
result RNNs need to learn attractors to fully use their power.
The study quantifies deep learning generalization error using data distribution and network smoothness.
problem Understanding the generalization error in deep learning models.
method Introducing cover complexity (CC) to measure data difficulty and using the inverse of the modulus of continuity to quantify neural network smoothness. A bound for expected accuracy/error is derived considering both CC and neural network smoothness.
result The expected error of trained neural networks scales with the square root of the number of classes and has a linear relationship with respect to the cover complexity.
TPBS models improve robustness to overfitting with localized Dirichlet energy regularization.
problem Global Dirichlet energy-based regularization fails for TPBS models due to perfect interpolation.
method Propose local Dirichlet energy regularization and two inference estimators.
result TPBS models outperform neural networks in overfitting regimes and maintain competitive performance otherwise.
Unified framework for adaptive connection sampling in GNNs improves performance and robustness.
problem Over-smoothing and over-fitting in deep GNNs.
method Adaptive connection sampling trained jointly with GNN model parameters.
result Adaptive connection sampling mathematically equivalent to Bayesian GNNs approximation.
Paper analyzes convergence of SGD in RNNs, proving linear convergence rate.
problem Analyzing convergence of SGD in multi-layer recurrent neural networks.
method Developed tools to analyze multi-layer networks with ReLU activations.
result SGD achieves linear convergence rate in training RNNs with sufficient neurons.
Paper proposes LCP for structural encodings, outperforming existing methods.
problem Improving Graph Neural Networks performance through effective structural encodings.
method Geometric perspective, Local Curvature Profiles (LCP) for structural encodings, combining with global positional encodings, comparing with rewiring techniques.
result LCP significantly outperforms existing structural encodings and combining LCP with global positional encodings improves performance.
The Gauss-Newton method is analyzed for neural networks using Riemannian optimization techniques.
problem Training neural networks with smooth activations and convergence rates.
method Riemannian optimization perspective, analyzing the Gauss-Newton method in both underparameterized and overparameterized regimes.
result Geometric convergence rates independent of conditioning and eigenvalues, demonstrating accelerated convergence.
A new SSL method improves medical image classification using global latent mixing.
problem Costly annotation of large-scale medical image data sets.
method Linear mixing of labeled and unlabeled data in both input and latent space.
result Improved performance in semi-supervised classification of thoracic disease and skin lesion.
Deep ReLU networks can approximate smooth functions nearly optimally.
problem Approximating smooth functions with deep neural networks.
method Using Taylor expansions and deep ReLU network approximations, the paper establishes optimal approximation error bounds.
result Deep ReLU networks of width and depth O ( N ln N ) \mathcal{O}(N\ln N) O ( N ln N ) and O ( L ln L ) \mathcal{O}(L\ln L) O ( L ln L ) can approximate f ∈ C s ( [ 0 , 1 ] d ) f\in C^s([0,1]^d) f ∈ C s ([ 0 , 1 ] d ) with an error O ( ∥ f ∥ C s ( [ 0 , 1 ] d ) N − 2 s / d L − 2 s / d ) \mathcal{O}(\|f\|_{C^s([0,1]^d)}N^{-2s/d}L^{-2s/d}) O ( ∥ f ∥ C s ([ 0 , 1 ] d ) N − 2 s / d L − 2 s / d ) . Deep neural networks with piecewise-polynomial activations can approximate smooth functions and their derivatives.
problem Approximating smooth functions and their derivatives with neural networks.
method Derives the depth, width, and sparsity required for approximation in Hölder norms.
result Deep neural networks with bounded weights can approximate Hölder smooth functions and their derivatives.
Deep networks can classify data on smooth curves with high probability.
problem Classifying data from two disjoint smooth curves on the unit sphere.
method Gradient descent on a deep neural network, proving convergence and generalization via NTK dynamics.
result Randomly-initialized gradient descent learns to classify all points on the two curves with high probability when the network is sufficiently deep.
The study links model generalization to the Hessian and Lipschitz constant.
problem Understanding and improving model generalization in neural networks.
method Connecting model generalization to the Hessian and Lipschitz constant, proposing a metric and optimization algorithm.
result Model generalization is related to the Hessian and Lipschitz constant, providing a new metric and optimization algorithm.
Complex-valued neural networks avoid spurious local minima.
problem Finding spurious local minima in neural networks.
method Proved no spurious local minima for shallow complex neural networks with quadratic activations.
result Complex-valued weights eliminate spurious local minima in neural networks.
Gradient descent converges linearly for overparameterized linear networks.
problem Convergence of gradient descent for overparameterized neural networks.
method Local Polyak-Lojasiewicz and Descent Lemma for overparameterized linear models.
result Gradient descent achieves linear convergence for two-layer linear networks under relaxed assumptions.
Smooth activations enable optimal error rates in neural networks for Sobolev function classes.
problem Achieving optimal approximation and estimation error rates for neural networks in Sobolev function classes.
method Study of neural networks with smooth activations, proving optimal rates via approximation and statistical properties.
result Constant-depth networks with smooth activations achieve optimal rates of approximation and estimation, demonstrating smoothness adaptivity.
Characterizes neural kernel and NNGP for various activations.
problem Understanding neural kernels and NNGP for non-RELU activations.
method Characterization of RKHS for various activation functions.
result Broad class of non-infinitely smooth activations generate equivalent RKHSs at different depths.
Neural network accuracy improves with denser training samples.
problem Improving neural network accuracy on unseen test samples.
method Bounding empirical training error smoothed across activation regions and using it to discard high-risk test samples.
result Discarding high-risk test samples based on error bounds improves prediction accuracy by up to 20%.
This thesis explores GNNs, categorizing them into local and global approaches.
problem Understanding the convergence of global GNNs and connecting local and global approaches.
method Categorization of GNNs into local and global, study of Invariant Graph Networks, connecting local and global approaches, and using local MPNN for graph coarsening.
result Established a connection between local and global GNN approaches.
D-VAE generates valid DAGs for neural architecture search and Bayesian network learning.
problem Generating valid DAGs for machine learning models.
method Proposes a novel DAG variational autoencoder (D-VAE) using graph neural networks and asynchronous message passing.
result Demonstrates the effectiveness of D-VAE through neural architecture search and Bayesian network structure learning.
Graph neural networks over-smooth when layers increase, reducing discriminative power.
problem Over-smoothing in graph neural networks reduces model performance as the number of layers increases.
method Analyzed over-smoothing in general graph neural network architecture using Dirichlet energy.
result The Dirichlet energy of embeddings converges to zero, leading to loss of discriminative power.
Proposes a differentiable LSE-ICNN for modeling multi-well potentials.
problem Modeling multi-well potentials in various scientific domains.
method Log-sum-exponential (LSE) mixture of input convex neural network (ICNN) modes.
result Smooth surrogate that retains convexity within basins and allows gradient-based learning.
Wide neural networks have non-attracting local minima.
problem Understanding the impact of suboptimal local minima in deep and wide neural networks.
method Construction of non-attracting local minima and saddle points in wide neural networks.
result Wide neural networks have non-attracting local minima, even though they are not negatively impacted by suboptimal local optima.
Deep neural networks with various activation functions can approximate Hölder smooth functions.
problem Expressivity of deep neural networks with general activation functions.
method Investigates approximation ability of deep neural networks with a broad class of activation functions, including Hölder smooth functions.
result Derives the required depth, width, and sparsity of deep neural networks to approximate Hölder smooth functions.
Gradient descent with random init solves 1HL NNs in under-param regime.
problem Learning a one-hidden-layer neural network with quadratic activations.
method Provable gradient-based method with random initialization.
result Gradient descent iterates converge to globally optimal model with linear rate.
Gradient descent converges linearly for neural networks with specific conditions.
problem Optimizing neural networks with fixed width and depth.
method Local Polyak-Lojasiewicz criterion for gradient flow and descent.
result Gradient descent converges to zero-loss solutions under certain conditions.
Small non-linearities in activation functions lead to bad local minima in neural networks.
problem Bad local minima in neural networks due to small non-linearities.
method Proved spurious local minima for ReLU(-like) networks and constructed counterexamples for more general activations.
result For almost all practical datasets, there exist infinitely many local minima in ReLU(-like) networks.
The paper provides recovery guarantees for CNNs with multiple kernels under polynomial sample and computational complexities.
problem Parameter recovery for non-overlapping CNNs with multiple kernels.
method Showed local strong convexity of squared loss for most popular activations, used tensor methods for initialization, and proved convergence of gradient descent.
result Gradient descent following tensor initialization converges to the global optimal with polynomial time complexity.
Spatial smoothing improves BNNs' accuracy, uncertainty, and robustness without increasing computational cost.
problem Large ensembles in BNNs increase computational cost and reduce performance.
method Spatial smoothing adds blur layers to convolutional neural networks to ensemble neighboring feature map points.
result Spatial smoothing improves BNNs' performance with fewer ensembles and enhances robustness.