New algorithm proves deep networks can learn better than shallow ones.
problem Understanding the power difference between shallow and deep neural networks.
method Identifying a class of Boolean functions and proving that logarithmic-depth networks can learn them efficiently using hierarchical reconstruction.
result First algorithmic separation between constant-depth and logarithmic-depth neural networks.
LdSM builds efficient multi-label decision trees with logarithmic depth.
problem Efficiently annotate data points with relevant subsets of labels from a large label set.
method Develops LdSM algorithm for multi-label decision trees with logarithmic depth, optimizing a novel objective function for balanced splits and high class purity.
result Minimizing the proposed objective function leads to pure and balanced data splits, achieving high prediction accuracy and low prediction time.
Estimates neural network error approximating compact sets.
problem Approximating compact subsets from Banach spaces with neural networks.
method Estimates error rates for neural networks of varying width and depth.
result Depth is crucial for better approximation rates, width alone does not improve.
This paper optimizes ReLU networks for approximating Hölder continuous functions.
problem Optimizing the approximation rate of ReLU networks in terms of width and depth.
method Constructive proof of ReLU networks' approximation power with specific width and depth constraints.
result Optimal approximation rate of ReLU networks with width and depth constraints.
This work improves the lottery ticket hypothesis by reducing over-parameterization requirement.
problem Approximating a neural network by pruning a randomly over-parameterized network.
method Connecting pruning ReLU networks to extsc{SubsetSum} problem, showing logarithmic over-parameterization sufficiency.
result Logarithmic over-parameterization is sufficient for approximating any target neural network.
Transformers can learn noisy linear systems with depth and IID data.
problem Learning noisy linear dynamical systems with transformers.
method Theoretical analysis of multi-layer and single-layer transformers with respect to L 2 L^2 L 2 -testing loss. result Single-layer transformers have a non-diminishing lower bound on approximation error, suggesting depth separation.
Logarithmic pruning simplifies lottery ticket hypothesis.
problem Finding efficient subnetworks in large neural networks.
method Logarithmic pruning approach to identify subnetworks.
result Randomly initialized subnetworks achieve comparable performance.
We consider the problem of estimating the conditional probability of a label in time O(log n), where n is the number of possible labels. We analyze a natural reduction of this problem to a set of binary regression problems organized in a tree structure, proving a regret bound that scales with the depth of the tree. Mot…
New algorithm speeds up sampling from logconcave densities.
problem Sampling from logconcave functions in statistics and ML.
method Solves ODEs to improve HMC and other sampling methods.
result Nearly linear runtime for polylogarithmic depth.
Smooth activations enable optimal error rates in neural networks for Sobolev function classes.
problem Achieving optimal approximation and estimation error rates for neural networks in Sobolev function classes.
method Study of neural networks with smooth activations, proving optimal rates via approximation and statistical properties.
result Constant-depth networks with smooth activations achieve optimal rates of approximation and estimation, demonstrating smoothness adaptivity.
New research limits how deep neural networks can be for certain functions.
problem Understanding the depth required for neural networks to represent specific functions.
method Mixed-integer optimization, polyhedral theory, tropical geometry.
result Neural networks with more than one layer are necessary to represent certain functions.
Deep networks with path norm regularization can approximate analytic functions.
problem Approximating analytic functions with neural networks.
method Path norm regularized deep networks with activation function.
result Deep networks can approximate analytic functions with logarithmic dependence on approximation error.
Deep ReLU networks can efficiently approximate Sobolev and Besov functions.
problem Approximating functions in Sobolev and Besov spaces using deep neural networks.
method Used deep ReLU neural networks with varied width and depth to approximate functions in Sobolev and Besov spaces.
result Generalized the approximation rate to hold under the Sobolev embedding condition.
In the context of tree-search stochastic planning algorithms where a generative model is available, we consider on-line planning algorithms building trees in order to recommend an action. We investigate the question of avoiding re-planning in subsequent decision steps by directly using sub-trees as action recommender. …
GCNs distinguish graph models based on embeddings, but depth matters.
problem GCNs distinguish between different random graph models.
method Investigated the power of GCNs of varying depths to distinguish between graph models.
result GCNs with logarithmic depth can distinguish certain graphons, but simpler architectures suffice for others.
Optimal ReLU networks can memorize any separable set of points with a small number of parameters.
problem The optimal number of parameters required to memorize a set of points using ReLU networks.
method Construction of ReLU networks with specific bit complexity to memorize points satisfying a mild separability assumption.
result Optimal ReLU networks can memorize any separable set of points with a number of parameters that is i l d e O ( N ) ilde{O}(\sqrt{N}) i l d e O ( N ) . Improved activation function NLReLU boosts neural network performance.
problem Performance issues with ReLU activation function.
method NLReLU uses parametric natural logarithmic transform to improve ReLU.
result NLReLU provides higher accuracy than ReLU in various neural networks.
Quantum circuits are hard to learn on average.
problem Learning the output distributions of quantum circuits is hard.
method Statistical query model analysis.
result Learning quantum circuits requires exponentially many queries.
New rule reduces exploration regret to logarithmic, improving bad episode handling.
problem Improving exploration regret in average reward MDPs.
method Replacing Doubling Trick with Vanishing Multiplicative rule in EVI-based algorithms.
result Regret is logarithmic under the new rule, significantly better than linear.
Deep residual networks trained with gradient descent have small generalization gap.
problem Limited theoretical understanding of why residual networks generalize well.
method Analyzing overparameterized deep residual networks trained by gradient descent.
result Demonstrates that residual networks have a small generalization gap between training and test error.
We introduce the notion of connection thickness of spheres in a Cayley graph, related to dead-ends and their retreat depth. It was well-known that connection thickness is bounded for finitely presented one-ended groups. We compute that for natural generating sets of lamplighter groups on a line or on a tree, connection…
Deep nets outperform shallow nets in complex feature realization.
problem Realizing complex data features with deep nets.
method Refined covering number estimates and analysis of approximation rates.
result Deep nets can improve performance without additional capacity costs for complex features.
Gradient descent optimally trains RNNs without overparameterization.
problem Training recurrent neural networks (RNNs) with gradient descent.
method Nonasymptotic analysis of gradient descent for RNNs with diagonal weight matrices.
result Gradient descent can achieve optimality in RNNs with a network size scaling logarithmically with the number of samples.
Deep neural networks with specific parameter sets can approximate smooth functions efficiently.
problem Approximating smooth functions with deep neural networks.
method Deep neural networks with ReLU activation and specific parameter sets { 0 , ± 1 2 , ± 1 , 2 } \{0,\pm \frac{1}{2}, \pm 1, 2\} { 0 , ± 2 1 , ± 1 , 2 } are used to approximate C β C_β C β -smooth functions. result The constructed networks can approximate C β C_β C β -smooth functions with parameters { 0 , ± 1 2 , ± 1 , 2 } \{0,\pm \frac{1}{2}, \pm 1, 2\} { 0 , ± 2 1 , ± 1 , 2 } efficiently, achieving the same convergence rate as sparse networks with parameters in [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] . A new depth measure for non-convex data supports, faster than halfspace depth.
problem Non-convex data supports in multivariate statistics.
method Extending halfspace depth to Reproducing Kernel Hilbert Space (RKHS).
result The new depth measure is consistent and can be computed faster.
The paper studies randomized approximations of Tukey's depth for log-concave isotropic data.
problem The challenge of approximating Tukey's depth in high dimensions.
method The study examines randomized algorithms for approximating Tukey's depth for log-concave isotropic data.
result Randomized algorithms correctly approximate maximal depth and close to zero depths but not intermediate depths.
Deep neural networks approximate functions in shift-invariant spaces with controlled error.
problem Approximating functions in shift-invariant spaces with neural networks.
method Using deep ReLU neural networks, estimating approximation error bounds based on network width and depth.
result Deep neural networks achieve optimal approximation rates for Sobolev spaces up to a logarithmic factor.
The paper analyzes generalization in deep contrastive learning.
problem Generalization analysis for unsupervised deep contrastive representation learning.
method Parameter-counting and norm-based bounds derived for neural networks of varying sizes and depths.
result Bounds are independent of network depth and size, reducing dependency on matrix norms.
This paper connects functional data analysis with machine learning techniques.
problem Lack of theoretical analysis for functional depths.
method Viewing functional depths as kernel mean embeddings in machine learning.
result Facilitates answers to open questions about functional depths.
Self-attention models benefit equally from width and depth, but beyond a certain point, depth becomes less efficient.
problem Understanding the optimal balance between depth and width in self-attention models.
method Theoretical predictions and empirical ablations on networks of varying depths and widths.
result An optimal width of 30K is recommended for a 1-Trillion parameter network, marking a significant width for self-attention models.
A new depth measure based on optimal control theory captures multi-modal data.
problem Statistical depths for high-dimensional data.
method Eikonal equations and optimal control theory.
result The new depth measure is robust under adversarial models.
A new depth measure and median defined on Hadamard manifolds.
problem Statistical depth and median on Hadamard manifolds.
method Horospherical depth and Busemann median defined using renormalized distance functions.
result The Busemann median exists for every Borel probability measure on Hadamard manifolds.
New approach uses loss functions to extend data depth for anomaly detection.
problem Anomaly detection in high-dimensional data.
method Introducing loss depths to generalize halfspace depth.
result New loss depths improve anomaly detection efficiency and interpretability.
Proves depth 2 neural networks can't approximate certain functions as well as depth 3 networks.
problem Approximating functions with depth 2 networks in high dimensions.
method Lower bound proof using worst-to-average-case random self-reducibility.
result Proves depth 2 networks can't approximate certain functions as well as depth 3 networks, resolving an open problem.
New findings on depth vs. width in neural networks, showing depth can improve learnability.
problem Understanding the role of depth in neural networks, especially when width is unbounded.
method Analyzing sample complexity for learnability in norm-controlled depth-2 and depth-3 ReLU networks.
result Depth can improve learnability of functions that are otherwise unlearnable with depth-2 networks.
Learning based methods have shown very promising results for the task of depth estimation in single images. However, most existing approaches treat depth prediction as a supervised regression problem and as a result, require vast quantities of corresponding ground truth depth data for training. Just recording quality d…
Introduces Polar Depth for analyzing multivariate heavy-tailed data extremes.
problem Analyzing the behavior of extremes from multivariate heavy-tailed distributions.
method Introduces Polar Depth, a novel statistical depth function expressed in polar coordinates.
result The polar depth of the largest observations converges to the polar depth of the limiting distribution as the threshold increases.
Study on feature learning dynamics in infinite-depth neural networks, focusing on ResNets.
problem Understanding how features evolve during training in deep neural networks, especially in the large-depth limit.
method Conditional Gaussian representations and SDE system with decoupled backward weights.
result Depth-induced suppression of forward-backward coupling in infinite-depth networks, leading to a decoupled forward-backward SDE system.
AutoGrow automatically discovers optimal depth in DNNs.
problem Designing optimal depth in deep neural networks is difficult and time-consuming.
method AutoGrow grows new layers in a seed architecture if it improves accuracy; stops if no improvement. Robust policies generalize to different architectures and datasets.
result AutoGrow discovers near-optimal depth on various datasets, improving accuracy-computation trade-off in ResNets.
Following the seminal idea of Tukey, data depth is a function that measures how close an arbitrary point of the space is located to an implicitly defined center of a data cloud. Having undergone theoretical and computational developments, it is now employed in numerous applications with classification being the most po…
Enhances SSL methods with depth cues for better image understanding.
problem Lack of depth cues in 2D image pixel maps limits SSL performance.
method Integrates depth signals from a pretrained monocular RGB-to-depth model into contrastive learning frameworks.
result Improves SSL methods' robustness and generalization with depth signals.
The paper proves barriers to approximating functions with small weights and depth in neural networks.
problem Proving barriers to approximating functions with constant depth neural networks.
method Reduction to open problems and natural-proof barriers in circuit complexity, and a new approach to polynomially-bounded functions.
result There are fundamental barriers to proving results beyond depth 4 for constant-depth neural networks.
Proposes a new method to estimate Bayesian neural network depth.
problem Estimating the depth of Bayesian neural networks.
method Uses a discrete truncated normal distribution to learn depth mean and variance, inferring posterior distributions by minimizing variational free energy.
result Improves test accuracy and reduces posterior depth variance on the spiral dataset.
seMCD computes depth functions with statistical guarantees using sequential Monte Carlo.
problem Computing depth functions is computationally challenging, especially in high dimensions.
method Sequential Monte Carlo methodology with theoretical and empirical guarantees.
result The seMCD method provides accurate depth approximations with fewer samples than traditional methods.
A new depth function improves multivariate data analysis by considering variability directions.
problem Developing a depth function that respects quantile properties and is affine-invariant.
method Integrating rank-weighted depth with affine-invariance and covariance matrices.
result The AI-IRW depth function provides accurate quantile estimates and is robust to data variability.
Data depth aids in identifying anomalies in multivariate data.
problem Detecting abnormal observations in multivariate datasets.
method Using data depth to assign abnormality labels to observations with lower depth values.
result Data depth effectively identifies anomalies in multivariate settings.
Proposes a new model to predict irrational customer behavior.
problem Irrational customer behavior in decision-making.
method Nonparametric choice model using decision trees and probability distributions.
result Decision forest model accurately predicts non-rational customer behavior.
Over-parameterized CNNs show U-shaped test risk with depth increase.
problem Understanding the impact of depth on test risk in over-parameterized CNNs.
method Empirical image classification experiments and linear regression framework.
result Test risk is U-shaped with increasing depth in over-parameterized CNNs.