Wide CNNs outperform infinite width networks, revealing scaling laws.
problem Understanding the performance difference between finite and infinite width convolutional networks.
method Diagrammatic approach to derive asymptotic width dependence for various quantities.
result The difference in performance between finite and infinite width models vanishes at a definite rate with respect to model width.
Wide networks with polynomial activations have proven asymptotic behavior.
problem Understanding the behavior of neural networks in the large width limit.
method Proving a conjecture for deep networks with polynomial activation functions.
result Tight bounds on the behavior of wide networks during stochastic gradient descent and derivation of their finite-width dynamics.
This paper studies large-width asymptotics for ReLU neural networks with α-Stable initializations.
problem Characterizing the large-width behavior of ReLU neural networks with α-Stable initializations.
method Analysis of the large-width distributions and training dynamics of ReLU neural networks initialized with α-Stable distributions.
result For ReLU neural networks with α-Stable initializations, the large-width training dynamics achieve zero training error at a linear rate, characterized by a random kernel.
Researchers create annular translators for mean curvature flow.
problem Constructing complete, properly embedded annular translators.
method Constructing a family of complete, properly embedded, annular translators in a slab.
result For each inner width \( b \ge \pi/2 \) and necksize \( s > 0 \), there exists a translator.
New insights into how depth and width affect in-context learning in deep models.
problem Understanding how various resources impact in-context learning in deep models.
method Analyzed linear regression in a deep linear self-attention model, varying resources like depth, width, context length, and training steps.
result Increasing depth improves in-context learning even at infinite context length, contrary to previous findings.
The paper addresses the gap between theoretical and practical confidence set widths in universal inference.
problem Inference procedures can be overly conservative, leading to wider confidence sets than expected.
method The authors identify the source of asymptotic conservativeness and propose a remedy based on studentization and bias correction.
result The proposed method achieves exact asymptotic coverage at the nominal 1−α level, even under model misspecification. Understanding the asymptotic behavior of wide networks is of considerable interest. In this work, we present a general method for analyzing this large width behavior. The method is an adaptation of Feynman diagrams, a standard tool for computing multivariate Gaussian integrals. We apply our method to study training dyn…
We study how finite Bayesian neural networks adapt their hidden representations.
problem Understanding how finite Bayesian neural networks differ from infinite ones.
method We analyze the asymptotics of learned feature kernels for various network architectures.
result The leading finite-width corrections to feature kernels have a universal form.
This paper analyzes deep Stable neural networks, showing convergence rates under different growth settings.
problem Analyzing the behavior of deep Stable neural networks as width increases.
method Large-width asymptotic analysis and convergence rates for fully connected feed-forward deep Stable NNs.
result The rescaled deep Stable NN converges weakly to a Stable SP under joint growth, with sup-norm convergence rates established.
We give asymptotically sharp upper bounds for the Khovanov width and the dealternation number of positive braid links, in terms of their crossing number. The same braid-theoretic technique, combined with Ozsváth, Stipsicz, and Szabó's Upsilon invariant, allows us to determine the exact cobordism distance between torus …
Study eigenvalue distributions of neural kernels for linear-width networks.
problem Eigenvalue distributions of neural kernels in linear-width networks.
method Asymptotic analysis of Conjugate Kernel and Neural Tangent Kernel under random initialization and approximate orthogonality.
result Eigenvalue distributions converge to deterministic limits, described by recursive fixed-point equations.
The paper studies deep neural networks with Gaussian weights and finds their asymptotic behavior.
problem Understanding the behavior of deep neural networks with large width.
method Function-space perspective, Gaussian process analysis, weak convergence in large-width limit.
result Deep neural networks with large width converge to a continuous Gaussian process.
Bayesian neural networks approximate Student-t processes in the infinite-width limit.
problem Modeling uncertainty in neural networks with greater flexibility.
method Extending asymptotic properties of Gaussian processes to Student-t processes in the infinite-width limit of BNNs.
result Posterior BNNs converge to Student-t processes in the infinite-width limit.
Define mass invariant for asymptotically hyperbolic 3-manifolds with toroidal infinity.
problem Define mass invariant for asymptotically hyperbolic 3-manifolds with toroidal infinity.
method Define mass invariant for asymptotically hyperbolic 3-manifolds with toroidal infinity.
result Define mass invariant for asymptotically hyperbolic 3-manifolds with toroidal infinity.
Sharp asymptotics reveal how network width controls learnability in quadratic neural networks.
problem Understanding learnability in overparameterized quadratic neural networks.
method Mapping ERM to convex matrix sensing with nuclear norm penalization.
result Characterization of global minima and precise generalization thresholds.
Bayesian linear networks reveal optimal depth and width trade-offs.
problem Understanding how depth, width, and dataset size affect model quality in linear networks.
method Zero noise Bayesian inference with Gaussian weight priors and mean squared error.
result Optimal predictions at infinite depth and maximized Bayesian model evidence at infinite depth.
The betting CI outperforms classical methods in constructing confidence intervals for bounded means.
problem Constructing nonasymptotic confidence intervals for bounded means.
method A betting-based approach to define and time-uniform variants of confidence intervals (CSs).
result The betting CI matches the fundamental limits, outperforming existing empirical Bernstein CIs.
We construct a compact, convex ancient solution of mean curvature flow in Rn+1 with O(1)×O(n) symmetry that lies in a slab of width π. We provide detailed asymptotics for this solution and show that, up to rigid motions, it is the only compact, convex, O(n)-invariant ancient solution that lies …
The paper proves unique geodesics on hyperbolic surfaces and finds lower bounds.
problem Characterizing geodesics on hyperbolic surfaces.
method One-parameter Allen-Cahn min-max constructions.
result Every geodesic occurs with multiplicity one and provides uniform sharp lower bounds.
The paper provides non-asymptotic Edgeworth expansions for neural network outputs.
problem Approximating deviations of finite-width neural networks from their Gaussian limit.
method Multidimensional Edgeworth expansions of arbitrary order for neural network outputs.
result Established a bound on the total variation distance between neural network output and its Edgeworth approximation.
Paper analyzes infinite-width attention layers using Tensor Programs.
problem Capturing the infinite-width limit of attention layers.
method Tensor Programs framework to rigorously identify the limit distribution.
result Derives exact form of infinite-width limit distribution without Gaussian approximations.
A Riemannian n-manifold M has k-dimensional Uryson width bounded by a constant c >0 if there exists a continuous map f from M to an k-dimensional polyhedral space P, such that the pullbacks f^{-1}(p) of all points p in P have diameters bounded by c. We prove that an n-dimensional Riemannian manifold M with at least n-k…
This paper establishes the (nearly) optimal approximation error characterization of deep rectified linear unit (ReLU) networks for smooth functions in terms of both width and depth simultaneously. To that end, we first prove that multivariate polynomials can be approximated by deep ReLU networks of width $\mathcal{O}(N…
Dropout and RaM become equivalent in large ResNets as depth and width increase.
problem Improving performance in deep learning models.
method Comparing Dropout and Random Gradient Masking in ResNets.
result Dropout and RaM converge to the same large-scale limiting dynamics in ResNets.
Characterizes test error in learning with deep, structured feature maps.
problem Characterizing test error in learning with deep, structured feature maps.
method Asymptotic analysis of feature covariance and population covariance.
result Closed-form formula for feature covariance in Gaussian rainbow neural networks.
Study of deep neural networks with dependent weights leading to new model limits and properties.
problem Characterizing deep neural networks with dependent weights in the infinite-width limit.
method Modeling weights as a mixture of Gaussian distributions and analyzing the infinite-width limit.
result Characterization of neural network layers by scalar parameters and Lévy measures, leading to new model limits.
New framework explains fast transfer of hyperparameters across model scales.
problem Understanding and optimizing hyperparameters for large-scale models.
method Developed a conceptual framework for HP transfer across scale, showing fast transfer is equivalent to useful transfer for compute-optimal grid search.
result Fast transfer of hyperparameters is equivalent to useful transfer for compute-optimal grid search, offering asymptotic computational advantage.
The study proves a tube theorem for complex hyperbolic manifolds.
problem Understanding the geometry of complex hyperbolic manifolds.
method Tubular neighborhood theorem and geometric combination theorem.
result Explicit estimates and bounds for tube widths in complex hyperbolic manifolds.
Paper characterizes gradient descent dynamics for neural networks with finite width.
problem Characterize gradient descent dynamics for multi-layer neural networks.
method Non-asymptotic state evolution theory for finite-width networks.
result Gradient descent dynamics provide precise distributional characterization.
Study on Bayesian deep linear networks with multiple outputs and convolutional layers.
problem Characterize feature learning in finite-width Bayesian deep linear networks.
method Exact and analytical formulas for joint and posterior distributions, using large deviation theory.
result Quantitative description of feature learning in infinite-width regime.
Study on critical points in random neural networks, revealing three regimes based on activation function.
problem Investigating the expected number of critical points in random neural networks.
method Deriving asymptotic formulas for critical points under infinite-width limit and suitable regularity conditions.
result Three distinct regimes of critical points behavior depending on activation function.
Sharp feature transitions revealed in extensive-width networks.
problem Learning hierarchical features from noisy queries in large networks.
method Information-theoretic analysis and heuristic decoupling argument.
result Sequential phase transitions in feature learnability and effective width.
Bayes-optimal learning of deep random networks with Gaussian weights is studied.
problem Learning a target function corresponding to a deep, extensive-width, non-linear neural network with random Gaussian weights.
method Closed-form expressions for Bayes-optimal test error, ridge regression, kernel and random features regression are computed.
result Optimally regularized ridge regression and kernel regression achieve Bayes-optimal performances, while logistic loss yields a near-optimal test error for classification.
The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.
problem Understanding training dynamics in deep linear networks from random initialization.
method Theoretical analysis of gradient descent dynamics in deep linear networks with random initialization and large data.
result Captures the 'wider is better' effect and hyperparameter transfer effects, contrasting with neural-tangent parameterization.
The study examines uniqueness and non-uniqueness of minimal surfaces in hyperbolic space.
problem Uniqueness and non-uniqueness of minimal surfaces in hyperbolic space.
method Analyzes criteria for uniqueness and constructs examples of non-uniqueness.
result Uniqueness of minimal surfaces is equivalent to uniqueness in a smaller class of stable minimal disks.
Proves a quantitative index theorem for positive scalar curvature metrics.
problem Studying conjectures and open questions on positive scalar curvature.
method Quantitative relative index theorem and λ-Lipschitz rigidity theorem. result Positive answers to Gromov's open questions on scalar curvature.
In this paper we prove a general and sharp Asymptotic Theorem for minimal surfaces in H2×R. As a consequence, we prove that there is no properly immersed minimal surface whose asymptotic boundary C is a Jordan curve homologous to zero in the asymptotic boundary of H2×R, say $\partial_\infty H^2\tim…
We consider the estimation of two-sample integral functionals, of the type that occur naturally, for example, when the object of interest is a divergence between unknown probability densities. Our first main result is that, in wide generality, a weighted nearest neighbour estimator is efficient, in the sense of achievi…
The study of random surfaces reveals asymptotic lengths of separating geodesics.
problem Understanding geometric properties of random hyperbolic surfaces.
method Analysis of Weil-Petersson measure and asymptotic behavior of lengths.
result The shortest separating closed geodesics have lengths about 2logg. We define the Wirtinger width of a knot. Then we prove the Wirtinger width of a knot equals its Gabai width. The algorithmic nature of the Wirtinger width leads to an efficient technique for establishing upper bounds on Gabai width. As an application, we use this technique to calculate the Gabai width of approximately …
Batch normalization makes deep neural networks' representations increasingly orthogonal.
problem Orthogonality of deep neural network representations.
method Random linear transformations in successive batch-normalizations.
result Orthogonality of representations improves SGD performance.
The paper constructs ancient solutions to curvature flows in bounded and unbounded regions.
problem Understanding ancient solutions to curvature flows in bounded and unbounded regions.
method Constructing pancake-like and sausage-like ancient compact solutions.
result Ancient solutions to curvature flows in bounded and unbounded regions.
We prove, in all dimensions n≥2, that there exists a convex translator lying in a slab of width πsecθ in Rn+1 (and in no smaller slab) if and only if θ∈[0,2π]. We also obtain convexity and regularity results for translators which admit appropriate symmetries and study the asymptotics a…
Empirical study compares finite- and infinite-width BNNs, revealing performance differences under model mismatch.
problem Comparing BNNs with different widths due to conflicting model properties and inference intractability.
method Empirical comparison of finite- and infinite-width BNNs, analyzing performance under model mismatch.
result Increasing width can hurt BNN performance when the model is mis-specified, and finite-width BNNs generalize better under model mismatch.
The isospectral problem for p-widths is solved using Zoll metrics on S^2.
problem Determine if a Riemannian manifold is uniquely determined by its p-widths.
method Construct counterexamples on S^2 using Zoll metrics and properties of geodesic p-widths.
result Many counterexamples exist on S^2, showing uniqueness is not guaranteed.
We propose a Generalized Dantzig Selector (GDS) for linear models, in which any norm encoding the parameter structure can be leveraged for estimation. We investigate both computational and statistical aspects of the GDS. Based on conjugate proximal operator, a flexible inexact ADMM framework is designed for solving GDS…
Width trees link link invariants and bridge number.
problem Understanding link invariants through geometric structures.
method Associate width trees to links and use their geometric properties to bound link invariants.
result Width trees uniquely realize certain link invariants under specific conditions.
Study of deep Stable neural networks with various activation functions.
problem Characterizing the infinitely wide limits of deep Stable neural networks.
method Investigation of large-width properties of deep Stable NNs with a generalized central limit theorem for heavy tails.
result Extension of characterization to a broader class of activation functions, including sub-linear, asymptotically linear, and super-linear functions.