Deep neural networks with adversarial training achieve sup-norm convergence for nonparametric regression.
problem Achieving sup-norm convergence for deep neural network estimators in nonparametric regression.
method Developed an adversarial training scheme to address the sup-norm convergence issue.
result Deep neural network estimators achieve optimal sup-norm convergence with the proposed adversarial training.
Proposes GTTN for discovering all low-rank structures in deep multi-task learning.
problem Discovering all low-rank structures among tasks in deep multi-task models.
method Introduces GTTN, a convex combination of matrix trace norms of all tensor flattenings, to automatically determine the importance of components.
result Demonstrates the effectiveness of GTTN on real-world datasets.
Efficiently regularizes deep learning models using Jacobian nuclear norm.
problem Regularizing deep learning models to prevent overfitting and improve generalization.
method Proposes a denoising-style approximation to penalize the Jacobian nuclear norm without computing the Jacobian matrix.
result Demonstrates that penalizing the average squared Frobenius norm of Jg and Jh is equivalent to penalizing the Jacobian nuclear norm for function compositions. Deep normative modeling of clinical neuroimaging data improves diagnostic performance.
problem Modeling variation of neuroimaging measures across individuals for psychiatric disorders.
method Proposes a deep normative modeling framework based on neural processes (NPs) for spatially structured mixed-effect modeling of neuroimaging data.
result Substantial improvements in novelty detection performance for certain diagnostic problems.
This work improves OOD detection using deep generative models by approximating Fisher information metrics.
problem Deep generative models often incorrectly infer higher likelihoods for out-of-distribution data.
method Approximating Fisher information metrics using gradient norms of data points.
result The method outperforms existing OOD detection techniques.
Deep networks with path norm regularization can approximate analytic functions.
problem Approximating analytic functions with neural networks.
method Path norm regularized deep networks with activation function.
result Deep networks can approximate analytic functions with logarithmic dependence on approximation error.
New proof shows norms can't explain deep learning's implicit regularization.
problem Understanding the implicit regularization in deep learning.
method Mathematical proof on matrix factorization problems.
result Implicit regularization drives norms towards infinity, suggesting rank minimization is key.
We investigate the generalizability of deep learning based on the sensitivity to input perturbation. We hypothesize that the high sensitivity to the perturbation of data degrades the performance on it. To reduce the sensitivity to perturbation, we propose a simple and effective regularization method, referred to as spe…
GAS-Norm improves deep learning time series forecasting in non-stationary settings.
problem Deep learning models struggle with non-stationary time series data.
method Combines GAS model for adaptive normalization with deep neural networks.
result Improves deep learning performance in 21 out of 25 settings.
Deep networks learn sparse hierarchical features without CoD.
problem Overparameterized deep networks struggle with the curse of dimensionality.
method Norm-constrained neural networks for sparse compositional functions.
result Deep networks can learn sparse hierarchical features efficiently.
We propose a new point of view for regularizing deep neural networks by using the norm of a reproducing kernel Hilbert space (RKHS). Even though this norm cannot be computed, it admits upper and lower approximations leading to various practical strategies. Specifically, this perspective (i) provides a common umbrella f…
Paper shows deep neural networks can approximate Korobov functions nearly optimally.
problem Approximating Korobov functions with deep neural networks.
method Used deep neural networks and measured approximation rates with Lp and H1 norms. result Achieved a super-convergence rate, outperforming traditional methods.
Deep linear networks can closely approximate interpolants without improving risk.
problem Understanding the risk bounds of deep linear networks compared to minimum ℓ2-norm solutions. method Bounding excess risk of interpolating deep linear networks trained using gradient flow.
result Deep linear networks can closely approximate or match minimum ℓ2-norm solutions in terms of risk. New bound relaxes uniform gradient norm assumptions for PAC-Bayesian bounds.
problem Generalization bounds with strict assumptions like uniformly bounded loss.
method Relax uniform bounds assumptions to on-average bounded loss and gradient norm.
result Proposes a new generalization bound with a surrogate of model complexity.
New capacity measure for deep ReLU networks derived from weight norms.
problem Identifying a suitable capacity measure for deep ReLU networks.
method Generalization of a recently proposed sampling argument to demonstrate the existence of sparse approximants of positive homogeneous networks.
result Bounding generalization error in multi-class classification using covering number bounds.
Adversarial training linked to operator norm regularization, proving network sensitivity to attacks.
problem Robustifying neural networks against adversarial attacks.
method Theoretical link established between adversarial training and operator norm regularization.
result Adversarial training is equivalent to data-dependent operator norm regularization.
Proposes a new generalization bound for Bayesian deep nets without strict assumptions.
problem Lack of generalization bounds for Bayesian deep nets without strict assumptions.
method Exploits contractivity of Log-Sobolev inequalities to add a loss-gradient norm term to the generalization bound.
result Introduces a new generalization bound for Bayesian deep nets that avoids strict assumptions.
Muon dynamics study uses spectral Wasserstein flow for optimization stability.
problem Optimizing deep learning models with gradient normalization.
method Introduces Spectral Wasserstein distances for matrix flows, proving equivalence with Benamou--Brenier formulation.
result Gradient-flow interpretation of mean-field normalized training dynamics.
Deep ResNets favor low bottleneck rank with proper hyperparameters.
problem Understanding the inductive bias of deep neural networks.
method Computed minimum-norm weights of a deep linear ResNet.
result Deep nonlinear ResNets have an inductive bias towards minimizing bottleneck rank.
Paper connects loss functions and t-norms for faster deep learning convergence.
problem Improving convergence rates in deep learning models.
method Interprets loss functions through t-norms and generator functions.
result Derives a general relation between loss functions and t-norms leading to faster convergence.
New bounds improve deep learning performance efficiently.
problem Improving generalization and robustness of deep learning models.
method Deriving four provable upper bounds on spectral norm of convolution layers, differentiable and efficient.
result Minimum of four bounds is a tight, differentiable and efficient upper bound on spectral norm.
Characterizes dropout's regularizer in deep linear networks.
problem Understanding dropout's regularization effect in deep learning.
method Formal characterization of dropout's regularizer, showing it is composed of an ℓ2-path regularizer and the squared nuclear norm. result For large dropout rates, the global optima of the dropout objective can be characterized.
Muon optimizer improves deep learning with spectral norm constraints.
problem Improving optimization algorithms in deep learning.
method Theoretical analysis of Muon optimizer within the Lion-K family. result Muon implicitly solves an optimization problem enforcing spectral norm constraints.
Paper finds exact Hessian sharpness in deep matrix factorization.
problem Understanding the geometry of loss landscapes in deep matrix factorization.
method Presented the first exact expression for Hessian maximum eigenvalue.
result Spectral-norm balance is a sufficient condition for flatness in deep matrix factorization.
New measure shows various training techniques control model complexity.
problem Understanding how to control model complexity in deep learning.
method Developed geometric complexity measure and demonstrated its effectiveness.
result Many training techniques control geometric complexity, providing a unified framework.
New algorithms estimate matrix norms without matrix multiplication.
problem Estimating matrix norms efficiently in a matrix-free setting.
method Randomized algorithms based on Hutchinson's estimator modifications.
result Oracle complexity bounds for two-to-infinity and one-to-two norms.
A novel metric and framework for evaluating gradient norm equality in deep neural networks.
problem Evaluation of gradient norm equality in complex DNNs requires strong assumptions or complex analysis.
method Proposes a novel metric called Block Dynamical Isometry and a modularized statistical framework based on free probability.
result Gradient Norm Equality is a universal philosophy behind initialization, normalization, and network structures.
SGD-trained deep nets have bounds on their generalization error.
problem Bounding generalization error for deep neural networks trained by SGD.
method Combining dynamical control of parameter norms and Rademacher complexity estimates.
result Explicit bounds depend on loss trajectory, work for various architectures.
Unified PAC-Bayesian framework for deep learning generalization.
problem Limitations of existing PAC-Bayesian norm-based bounds for deep neural networks.
method Unified framework using anisotropic Gaussian posteriors and sensitivity matrix.
result Comparable or tighter generalization bounds compared to state-of-the-art approaches.
Deep ReLU networks can approximate matrix-vector products with error bounds.
problem Can deep ReLU networks accurately approximate matrix-vector products?
method Derived error bounds in Lebesgue and Sobolev norms for deep ReLU FNNs.
result Developed deep approximation theory with successful applications.
New bounds for neural networks ensure robustness and accuracy.
problem Ensuring robustness of neural networks by computing Lipschitz constants.
method Analyzed and proposed new bounds for l1 and l∞ norms, using explicit and implicit methods for convnets. result One of the new bounds is optimal and more accurate than existing ones.
Deep neural networks (DNNs) have become increasingly important due to their excellent empirical performance on a wide range of problems. However, regularization is generally achieved by indirect means, largely due to the complex set of functions defined by a network and the difficulty in measuring function complexity. …
Deep belief networks can approximate any multivariate density with binary hidden units.
problem Approximating multivariate probability densities with binary hidden units.
method Sharp quantitative bounds on approximation error in terms of hidden units.
result Deep belief networks can approximate any multivariate density with binary hidden units under mild integrability requirements.
AMP regularization improves deep learning models by favoring flat minima.
problem Improving deep learning model generalization and avoiding overfitting.
method AMP regularization uses adversarial model perturbation to minimize a norm-bounded perturbation of the empirical risk.
result AMP regularization leads to state-of-the-art performance across various deep architectures.
ResNets minimize circuit size for fitting data in HTMC regime.
problem Finding the simplest algorithm that fits data.
method Defining HTMC and ResNet norms to relate circuit size and function fitting.
result Minimizing ResNet norm is equivalent to finding a circuit with minimal nodes.
Fine-tunes deep neural networks to match theoretical bounds on generalization errors.
problem Improve generalization errors of deep neural networks by constraining weight norms.
method Proposes a two-stage renormalization procedure and a fine-grained SGD algorithm for training DNNs with constrained weights.
result Empirical generalization errors of DNNs are closer to theoretical bounds, improving accuracy.
Theoretical justification for deep networks' performance with regularization techniques.
problem Understanding the performance of deep networks trained with the square loss.
method Analysis of gradient flow and theoretical justification of regularization techniques.
result Convergence to solutions with smaller Frobenius norms leads to better classification error bounds.
Proposes model-based robust deep learning to handle natural variation in data.
problem Deep learning's fragility to natural variation in data.
method Develops model-based robust training algorithms using deep generative models to learn natural variation.
result Deep neural networks trained with model-based algorithms outperform standard and norm-bounded robust algorithms.
The paper analyzes generalization in deep contrastive learning.
problem Generalization analysis for unsupervised deep contrastive representation learning.
method Parameter-counting and norm-based bounds derived for neural networks of varying sizes and depths.
result Bounds are independent of network depth and size, reducing dependency on matrix norms.
A new method bridges explicit and implicit deep generative models using Stein discrepancy.
problem Limitations of explicit and implicit deep generative models.
method Joint training framework that combines an explicit density estimator and an implicit sample generator via Stein discrepancy.
result The method improves the accuracy of density estimation and quality of generated samples.
New properties of deep ReLU networks at initialization improve performance.
problem Improving performance of deep ReLU networks.
method PAC analysis framework to prove novel properties of He initialization.
result Hidden activation norms and weight gradient norms are preserved under He initialization.
Exact spectral norm regularization improves neural network generalization.
problem Improving neural network generalization while protecting against noise.
method Exact spectral norm regularization of the Jacobian.
result Improved generalization performance compared to previous methods.
Deep neural solvers can approximate harmonic functions with low error.
problem Harmonic functions in Barron space are not regular.
method Elliptic regularity theory applied to Barron functions.
result Approximation of harmonic functions by Barron functions with low error.
Unified theory of deep neural networks with diverse activations.
problem Understanding the relationship between depth and complexity in deep neural networks.
method Developed a unified function space theory for deep networks with various activations.
result Unified theory provides meaningful complexity for deep networks with diverse activations.
Complexity measures for neural nets with general activations using path-based norms.
problem Control complexity of neural networks with arbitrary activation functions.
method Approximate general activations with ReLU networks and derive path-based norms for complexity control.
result Preliminary analyses of function spaces and regularized estimators.
Gradient flow with weight decay shows grokking effect in deep learning.
problem Understanding the grokking effect in deep learning.
method Analyzing gradient flow dynamics with weight decay.
result Weight decay causes slow norm reduction, explaining grokking.
We propose an adaptive optimization method for deep learning that dynamically adjusts batch size.
problem Optimizing deep learning models with varying sensitivity to batch size selection.
method Adaptive regularization with dynamically determined stochastic batch size based on gradient norms.
result Our method outperforms state-of-the-art optimization algorithms in generalization and robustness.
The paper analyzes and improves a deep learning optimization technique using matrix gradient orthogonality.
problem Improving deep learning training through more effective optimization methods.
method Develops a stochastic non-Euclidean trust-region gradient method for deep learning optimization.
result Proves state-of-the-art convergence results for the proposed algorithm in various scenarios.