Empirical study compares finite- and infinite-width BNNs, revealing performance differences under model mismatch.
problem Comparing BNNs with different widths due to conflicting model properties and inference intractability.
method Empirical comparison of finite- and infinite-width BNNs, analyzing performance under model mismatch.
result Increasing width can hurt BNN performance when the model is mis-specified, and finite-width BNNs generalize better under model mismatch.
New framework for understanding infinite-width neural networks.
problem Understanding the infinite-width limit behavior of neural networks.
method General framework to study limit behavior of neural models based on hyperparameter scaling.
result Derives scaling for existing mean-field and neural tangent kernel limits and introduces new dynamically stable limits.
Wide neural networks can degrade performance, contrary to conventional wisdom.
problem Understanding the limitations of increasing network width in neural networks.
method Using Deep Gaussian Processes to decouple capacity and width, analyzing their effects on representational power and non-Gaussianity.
result Wide neural networks can become less adaptable and more Gaussian, leading to performance degradation.
Lectures on deep learning properties in infinite and large-width networks.
problem Understanding deep neural networks in extreme width conditions.
method Analysis of random deep neural networks, connections to linear models, kernels, and Gaussian processes, perturbative and non-perturbative treatments.
result Properties and behaviors of deep neural networks in the infinite-width limit and large-width regime.
Wide CNNs outperform infinite width networks, revealing scaling laws.
problem Understanding the performance difference between finite and infinite width convolutional networks.
method Diagrammatic approach to derive asymptotic width dependence for various quantities.
result The difference in performance between finite and infinite width models vanishes at a definite rate with respect to model width.
Wider neural networks perform better than deeper ones with the same number of parameters.
problem Understanding the role of network width versus the number of parameters in neural network performance.
method Comparing models with different ways of increasing width while keeping the number of parameters constant, analyzing their performance and using Gaussian Process kernels for analysis.
result Network width is the determining factor for good performance, while the number of weights is secondary as long as trainability is ensured.
Fisher width is a geometric measure of complexity on statistical manifolds.
problem Complexity measures on statistical manifolds
method Introducing Fisher width as a Fisher-geometric analogue of Gaussian width
result Fisher width retains key structural features of Gaussian width while capturing anisotropic geometric effects
Self-attention models benefit equally from width and depth, but beyond a certain point, depth becomes less efficient.
problem Understanding the optimal balance between depth and width in self-attention models.
method Theoretical predictions and empirical ablations on networks of varying depths and widths.
result An optimal width of 30K is recommended for a 1-Trillion parameter network, marking a significant width for self-attention models.
We define the Wirtinger width of a knot. Then we prove the Wirtinger width of a knot equals its Gabai width. The algorithmic nature of the Wirtinger width leads to an efficient technique for establishing upper bounds on Gabai width. As an application, we use this technique to calculate the Gabai width of approximately …
New optimizers control network width scaling, improving stability and transfer across different model sizes.
problem Designing stable optimizers for networks of varying widths.
method Interpreting optimizers as steepest descent under mean-normalized operator norms, enabling layerwise composability and width-independent bounds.
result New optimizers like row normalization and column normalization provide stable learning-rate transfer across different model widths.
Unified spectral framework for μP under joint width-depth scaling.
problem Challenges in stable feature learning and HP transfer for width-depth scaled models.
method Developed a simple and unified spectral framework for μP under joint width-depth scaling.
result Unified and generalized μP formulation for practical architectures with multi-transformation branches.
Joslim optimizes both width and weight configurations for slimmable neural networks, improving model efficiency.
problem Optimizing both width and weight configurations for slimmable neural networks to improve efficiency.
method Proposes a general framework for joint optimization of width configurations and weights, and introduces Joslim algorithm.
result Improves model efficiency by up to 1.7% in top-1 accuracy on the ImageNet dataset.
In this paper we adopted state-of-the-art machine learning algorithms, namely: random forest (RF) and least squares boosting, to model crash data and identify the optimum model to study the impact of narrow lanes on the safety of arterial roads. Using a ten-year crash dataset in four cities in Nebraska, two machine lea…
The isospectral problem for p-widths is solved using Zoll metrics on S^2.
problem Determine if a Riemannian manifold is uniquely determined by its p-widths.
method Construct counterexamples on S^2 using Zoll metrics and properties of geodesic p-widths.
result Many counterexamples exist on S^2, showing uniqueness is not guaranteed.
Width trees link link invariants and bridge number.
problem Understanding link invariants through geometric structures.
method Associate width trees to links and use their geometric properties to bound link invariants.
result Width trees uniquely realize certain link invariants under specific conditions.
Develops a new theory for neural systems stability and width effects.
problem Stability and finite-width effects in deep neural systems.
method Gauge-covariant stochastic effective field theory using classical commuting fields.
result Predicts the edge of chaos and low-frequency spectral deformation.
Computed p-widths for hemisphere, first for manifolds with boundary.
problem Finding p-widths for manifolds with boundary.
method Computed p-widths for the hemisphere.
result First known p-widths for a manifold with boundary.
Polygon p-widths are found via billiard trajectories.
problem Finding p-widths of polygons. method Proved via billiard trajectories and computed specific cases.
result Polygon p-widths are achieved by billiard trajectories. A number of results for C2-smooth surfaces of constant width in Euclidean 3-space E3 are obtained. In particular, an integral inequality for constant width surfaces is established. This is used to prove that the ratio of volume to cubed width of a constant width surface is reduced by shrinking it along…
Computed p-widths for real projective plane.
problem Calculating p-widths for real projective plane.
method Standard metric used to compute p-widths.
result Computed p-widths for real projective plane.
Study bounds Urysohn width of manifolds under surgeries.
problem Bounding Urysohn width of manifolds after surgeries.
method Analyzes connected sums and universal covers, applies to general surgeries.
result Optimal constants in estimates of width bounds are shown.
Residual networks with block width max(d_x, d_y) approximate all functions.
problem Achieving universal approximation with residual networks.
method Established bounds on block width for different activation functions.
result Minimum block width for universal approximation is max(d_x, d_y) with inner width 1.
Bayesian neural networks approximate Student-t processes in the infinite-width limit.
problem Modeling uncertainty in neural networks with greater flexibility.
method Extending asymptotic properties of Gaussian processes to Student-t processes in the infinite-width limit of BNNs.
result Posterior BNNs converge to Student-t processes in the infinite-width limit.
Empirical study shows standard CNNs deviate from NTK predictions.
problem Understanding how standard finite-width CNNs behave compared to their infinite-width NTK counterparts.
method Empirical analysis of AlexNet and LeNet architectures.
result Standard CNNs deviate significantly from their NTK counterparts, but deviation decreases with wider networks.
Deep neural networks' infinite-width behavior approximated by Gaussian models.
problem Understanding the behavior of deep neural networks in the limit of infinite width.
method Using the Lindeberg exchange principle to approximate weights by Gaussian random variables.
result Quantitative bounds on the 2-Wasserstein distance between deep neural networks and Gaussian limits.
While studying the existence of closed geodesics and minimal hypersurfaces in compact manifolds, the concept of width was introduced in different contexts. Generally, the width is realized by the energy of the closed geodesics or the volume of minimal hypersurfaces, which are found by the Minimax argument. Recently, Ma…
New Transformer architecture prevents rank degeneracy in deep attention models.
problem Rank degeneracy in deep attention models.
method Modified Softmax-based attention model with skip connections, centered at identity, and scaled logits.
result Existence of a stable SDE implies well-behaved covariance structure, preventing rank degeneracy.
Study Gaussian-process limits of neural networks using tensor programs.
problem Understanding the behavior of neural networks as they approach infinite width.
method Quantitative analysis through tensor programs and Wasserstein distance.
result Explicit finite-width error bounds, showing convergence to Gaussian-process limits.
Stochastic neural networks with infinite width become deterministic, reducing training variance.
problem Understanding how stochasticity in neural networks affects learning and regularization.
method Theoretical analysis of stochastic neural networks with infinite width.
result As the width of an optimized stochastic neural network increases, its predictive variance on the training set decreases to zero.
Proves conjecture about sphere widths under rotational symmetry.
problem Width stability of rotationally symmetric metrics.
method Proof of conjecture and extensions to higher dimensions.
result Stability of min-max width under rotational symmetry.
New link invariants from diagram colorings match link widths.
problem Defining link widths via diagram colorings.
method Colorings of link diagrams to define invariants and prove their equivalence to link widths.
result Invariants of link widths calculated algorithmically.
Study infinite-depth limits of neural networks with fixed width.
problem Understanding the behavior of neural networks as depth increases with fixed width.
method Analyzing finite-width residual networks with random Gaussian weights, focusing on the infinite-depth limit.
result The pre-activations converge to a zero-drift diffusion process, differing from the infinite-width limit.
Novel algorithms improve efficiency of finite width NTK computation.
problem Efficiency of computing finite width Neural Tangent Kernel (NTK).
method Leveraging neural network structure, propose two novel algorithms.
result Significantly improved efficiency in compute and memory requirements.
In "Width complexes for knots and 3-manifolds," Jennifer Schultens defines the width complex for a knot in order to understand the different positions a knot can occupy in the 3-sphere and the isotopies between these positions. She poses several questions about these width complexes; in particular, she asks whether the…
Sharp lower bound for first Neumann eigenvalue found in terms of diameter and width.
problem Finding the minimum value of the first Neumann eigenvalue for convex domains.
method Proved the sharp lower bound using diameter and width.
result Sharp lower bound for the first Neumann eigenvalue established.
We prove that among all constant width bodies of revolution, the minimum of the ratio of the volume to the cubed width is attained by the constant width body obtained by rotation of the Reuleaux triangle about an axis of symmetry.
Develops a new theory of width for embedded circles in Riemannian manifolds.
problem Defining and understanding the width of embedded circles in Riemannian manifolds.
method Morse-Lusternik-Schnirelmann theory applied to geodesics and minimising configurations.
result Classifies configurations of minimising geodesics intersecting embedded circles.
This work optimizes neural network bit-width and layer-width for efficiency.
problem Efficient optimization of deep neural networks for reduced size and computational demands.
method Cluster-based tree-structured Parzen estimator for surrogate modeling, Hessian-based pruning for parameter reduction.
result 20% decrease in model size with 12x reduction in search time compared to existing methods.
The study reveals a transition in neural network performance from infinite-width to variance-limited behavior as dataset size increases.
problem Understanding the transition from infinite-width to variance-limited behavior in neural networks.
method Empirical study of the transition from infinite-width to variance-limited behavior as a function of sample size and network width.
result The critical sample size \( P^* \) is approximately \( \sqrt{N} \) for polynomial regression with ReLU networks.
This paper explores how neural network width and depth behave as they approach infinity.
problem Understanding the behavior of neural functions as width and depth go to infinity.
method Formal definition of commutativity framework, study of neural covariance kernel, novel proof techniques.
result Taking width and depth to infinity in a deep neural network with skip connections results in the same covariance structure, regardless of the order of taking limits.
Paper proves a noncompact version of Gromov's band-width estimate.
problem Proving a precise upper bound for noncompact Riemannian bands.
method Developed a quantitative partitioned manifold index theory.
result Proved a version of Gromov's band-width estimate for noncompact Riemannian bands.
We discuss a possible definition for "k-width" of both a closed d-manifold Md, and on embedding Md↪eRn, n>d≥k, generalizing the classical notion of width of a knot. We show that for every 3-manifold 2-width(M3)≤2 but that there are embeddings $e_i: T^3 \hoo…
The study examines spectral dynamics in deep neural networks, predicting how outliers evolve during training.
problem Understanding spectral evolution in deep neural networks during training.
method Developed a two-level dynamical mean-field theory (DMFT) to track spectral dynamics.
result The theory predicts how outliers evolve with training time, width, output scale, and initialization variance.
We extend the classical definition of {\it width} to higher dimensional, smooth codimension 2 knots and show in each dimension there are knots of arbitrarily large width.
Paper characterizes gradient descent dynamics for neural networks with finite width.
problem Characterize gradient descent dynamics for multi-layer neural networks.
method Non-asymptotic state evolution theory for finite-width networks.
result Gradient descent dynamics provide precise distributional characterization.
We present an alternative proof of the following fact: the hyperspace of compact closed subsets of constant width in Rn is a contractible Hilbert cube manifold. The proof also works for certain subspaces of compact convex sets of constant width as well as for the pairs of compact convex sets of constant rela…
Study examines dependence properties of Bayesian neural network units in finite-width networks.
problem Understanding dependence properties of hidden units in practical finite-width Bayesian neural networks.
method Theoretical analysis and empirical evaluation of depth and width impacts.
result Hidden units in finite-width Bayesian neural networks are dependent, contrary to the infinite-width limit assumption.
We study surface knots in 4-space by using generic planar projections. These projections have fold points and cusps as their singularities and the image of the singular point set divides the plane into several regions. The width (or the total width) of a surface knot is a numerical invariant related to the number of po…