On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,657 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “Width Scaling”

Robust Layerwise Scaling Rules by Proper Weight Decay Tuningcs.LG

We introduce a new weight-decay scaling rule to maintain sublayer gains across different widths in modern scale-invariant architectures.

problem In modern scale-invariant architectures, training quickly enters a steady state where normalization layers create backward scale sensitivity, degrading learning-rate transfer.

method We introduce a weight-decay scaling rule for AdamW that preserves sublayer gain across widths by equalizing the effective learning rate.

result Our empirical weight-decay scaling rule

λ_2\propto \sqrt{d}

approximately keeps sublayer gains width invariant, enabling zero-shot transfer of learning rate and weight decay.

weight decay sublayer gain learning rate scale-invariant architectures AdamW

2025-10-17abs ↗pdf ↗

Dynamically Stable Infinite-Width Limits of Neural Classifierscs.LG

New framework for understanding infinite-width neural networks.

problem Understanding the infinite-width limit behavior of neural networks.

method General framework to study limit behavior of neural models based on hyperparameter scaling.

result Derives scaling for existing mean-field and neural tangent kernel limits and introduces new dynamically stable limits.

neural networks infinite width mean-field neural tangent kernel hyperparameter scaling

2020-06-11abs ↗pdf ↗

Spectral Condition for

μ

P under Width-Depth Scalingcs.LG

Unified spectral framework for μP under joint width-depth scaling.

problem Challenges in stable feature learning and HP transfer for width-depth scaled models.

method Developed a simple and unified spectral framework for μP under joint width-depth scaling.

result Unified and generalized μP formulation for practical architectures with multi-transformation branches.

μP spectral framework width-depth scaling residual networks language models

2026-02-28abs ↗pdf ↗

Asymptotics of Wide Convolutional Neural Networkscs.LG

Wide CNNs outperform infinite width networks, revealing scaling laws.

problem Understanding the performance difference between finite and infinite width convolutional networks.

method Diagrammatic approach to derive asymptotic width dependence for various quantities.

result The difference in performance between finite and infinite width models vanishes at a definite rate with respect to model width.

wide neural networks convolutional neural networks asymptotics scaling laws skip connections

2020-08-19abs ↗pdf ↗

On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfercs.LG

New optimizers control network width scaling, improving stability and transfer across different model sizes.

problem Designing stable optimizers for networks of varying widths.

method Interpreting optimizers as steepest descent under mean-normalized operator norms, enabling layerwise composability and width-independent bounds.

result New optimizers like row normalization and column normalization provide stable learning-rate transfer across different model widths.

Neural Optimizers Width Scaling Operator Norms Steepest Descent Layerwise Composability

2026-03-10abs ↗pdf ↗

Commutative Width and Depth Scaling in Deep Neural Networksstat.ML

This paper explores how neural network width and depth behave as they approach infinity.

problem Understanding the behavior of neural functions as width and depth go to infinity.

method Formal definition of commutativity framework, study of neural covariance kernel, novel proof techniques.

result Taking width and depth to infinity in a deep neural network with skip connections results in the same covariance structure, regardless of the order of taking limits.

Neural Networks Depth and Width Scaling Commutativity Neural Covariance Kernel Skip Connections

2023-10-02abs ↗pdf ↗

Subquadratic Overparameterization for Shallow Neural Networkscs.LG

New method trains shallow neural networks with subquadratic width scaling.

problem Training shallow neural networks with optimal width scaling.

method Polyak-Lojasiewicz condition, smoothness, standard data assumptions, random matrix theory.

result Subquadratic scaling on network width with standard initialization strategies.

neural networks overparameterization shallow networks subquadratic scaling Polyak-Lojasiewicz

2021-11-02abs ↗pdf ↗

On the Surprising Effectiveness of Large Learning Rates under Standard Width Scalingcs.LG

Large learning rates work surprisingly well in standard parameterization, contrary to theory.

problem Theoretical limits of large learning rates do not match practical network behavior.

method Fine-grained analysis of learning rates and network behavior under cross-entropy loss.

result There are two distinct sub-regimes of unstable learning rates, with a controlled divergence regime where features continue to evolve.

large learning rates standard parameterization cross-entropy loss finite-width phenomena infinite-width limits

2025-05-28abs ↗pdf ↗

Meta-Principled Family of Hyperparameter Scaling Strategiescs.LG

Derives a family of hyperparameter scaling strategies for neural networks.

problem Optimizing hyperparameters for wide and deep neural networks.

method Introduces a one-parameter family of hyperparameter scaling strategies.

result Reveals proper scaling of depth with width for large-scale models.

hyperparameter scaling neural networks scaling strategies neural tangent kernels wide and deep networks

2022-10-10abs ↗pdf ↗

Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networksstat.ML

Sharp feature transitions revealed in extensive-width networks.

problem Learning hierarchical features from noisy queries in large networks.

method Information-theoretic analysis and heuristic decoupling argument.

result Sequential phase transitions in feature learnability and effective width.

feature learning Bayes-optimal scaling laws high-dimensional networks

2026-05-11abs ↗pdf ↗

Width and Depth Limits Commute in Residual Networksstat.ML

Theoretical limits of deep residual networks show consistent covariance structures.

problem Understanding the limits of deep residual networks.

method Analyzing the behavior of deep residual networks with skip connections as width and depth approach infinity.

result Theoretical analysis confirms that the covariance structure remains consistent regardless of the order of width and depth.

residual networks deep learning covariance structure skip connections infinite width

2023-02-01abs ↗pdf ↗

Explaining Neural Scaling Lawscs.LG

Theory explains neural network scaling with dataset and model size.

problem Neural network scaling laws with dataset and model size.

method Identified variance-limited and resolution-limited scaling behaviors.

result Four scaling regimes explained: infinite data, infinite width, resolution-limited, and large width.

neural networks scaling laws dataset size model size infinite data

2021-02-12abs ↗pdf ↗

Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programscs.LG

Paper analyzes infinite-width attention layers using Tensor Programs.

problem Capturing the infinite-width limit of attention layers.

method Tensor Programs framework to rigorously identify the limit distribution.

result Derives exact form of infinite-width limit distribution without Gaussian approximations.

attention layers Tensor Programs infinite-width limit non-Gaussianity Transformer architectures

2025-06-01abs ↗pdf ↗

Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitstat.ML

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

problem The challenge of hyperparameter tuning in deep learning, especially for large models.

method Combining

μ

P parameterization with residual networks having a residual branch scale of

1/\sqrt{ ext{depth}}

result Optimal hyperparameters transfer across width and depth in residual networks trained with this parameterization.

hyperparameter tuning residual networks depthwise scaling mean field theory neural network dynamics

2023-09-28abs ↗pdf ↗

The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Studycs.LG

We investigate how the final parameters found by stochastic gradient descent are influenced by over-parameterization. We generate families of models by increasing the number of channels in a base network, and then perform a large hyper-parameter search to study how the test error depends on learning rate, batch size, a…

2019-05-09abs ↗pdf ↗

Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Timestat.ML

New insights into how depth and width affect in-context learning in deep models.

problem Understanding how various resources impact in-context learning in deep models.

method Analyzed linear regression in a deep linear self-attention model, varying resources like depth, width, context length, and training steps.

result Increasing depth improves in-context learning even at infinite context length, contrary to previous findings.

in-context learning deep learning scaling laws transformer models linear regression

2025-10-01abs ↗pdf ↗

How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterizationcs.LG

New scaling framework for MoE architectures ensures stability and optimal performance at scale.

problem Lack of principled understanding of how hyperparameters should scale in MoE architectures.

method Developed a novel Dynamical Mean Field Theory (DMFT) for three scaling regimes of MoE architectures.

result Derived Maximally Scale-Stable Parameterization (MSSP) for SGD and Adam, providing robust learning rate transfer and monotonic improvement with scale.

Mixture-of-Experts Scaling Maximal-scale stability Dynamical Mean Field Theory MoE architectures

2026-05-13abs ↗pdf ↗

Understanding the Mechanisms of Fast Hyperparameter Transfercs.LG

New framework explains fast transfer of hyperparameters across model scales.

problem Understanding and optimizing hyperparameters for large-scale models.

method Developed a conceptual framework for HP transfer across scale, showing fast transfer is equivalent to useful transfer for compute-optimal grid search.

result Fast transfer of hyperparameters is equivalent to useful transfer for compute-optimal grid search, offering asymptotic computational advantage.

hyperparameter optimization scale-aware HPs fast transfer compute efficiency Maximal Update Parameterization

2025-12-28abs ↗pdf ↗

Width-Robust Learnability in Mean-Field Bayesian Neural Networksstat.ML

Bayesian neural networks learn efficiently at infinite width, matching polynomial-width performance.

problem Understanding the inductive bias of infinite-width neural networks.

method Analyzing the reduced entropy and using subsampling techniques.

result The Bayesian mean-field learner generalizes exactly on polynomially-bounded targets.

Bayesian neural networks mean-field infinite width reduced entropy subsampling

2026-07-07abs ↗pdf ↗

How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networkscs.LG

Study how generalization scales with model size and data in quadratic neural networks.

problem Understanding how generalization scales with model size and data in quadratic neural networks.

method Analyzed

\ell_2

-regularized empirical test error minimization in a quadratic two-layer network with finite-sample setting and structured data.

result Revealed a phase diagram with distinct scaling regimes as the number of parameters varies, showing data-dependent power laws controlled by spectral structure of the target.

Quadratic Neural Networks Generalization Scaling Laws Data-Dependent Power Laws Spectral Structure

2026-06-26abs ↗pdf ↗

On the infinite-depth limit of finite-width neural networksstat.ML

Study infinite-depth limits of neural networks with fixed width.

problem Understanding the behavior of neural networks as depth increases with fixed width.

method Analyzing finite-width residual networks with random Gaussian weights, focusing on the infinite-depth limit.

result The pre-activations converge to a zero-drift diffusion process, differing from the infinite-width limit.

neural networks infinite-depth finite-width residual networks random weights

2022-10-03abs ↗pdf ↗

CLAPS: Aleatoric-Epistemic Scaling via Last-Layer Laplace for Conformal Regressioncs.LG

CLAPS improves conformal regression by adaptively scaling interval widths based on last-layer Laplace uncertainty.

problem Lack of adaptive interval width scaling in conformal regression for heterogeneous inputs.

method CLAPS uses heteroscedastic last-layer Laplace uncertainty to adaptively scale interval widths, combining aleatoric and epistemic uncertainties.

result CLAPS provides competitive interval efficiency with nominal-level coverage, reducing to aleatoric scaling as epistemic uncertainty decreases.

conformal regression interval scaling aleatoric noise epistemic uncertainty heteroscedasticity

2025-12-01abs ↗pdf ↗

A Dynamical Model of Neural Scaling Lawsstat.ML

Model predicts neural network performance scaling laws across various factors.

problem Understanding the performance of neural networks across different training factors.

method Random feature model trained with gradient descent, analyzing compute-optimal scaling laws.

result Predicts asymmetric compute-optimal scaling rule and behavior of training and test loss gap.

neural networks scaling laws compute-optimal training time model size

2024-02-02abs ↗pdf ↗

Spectral Dynamics in Deep Networks: Feature Learning, Outlier Escape, and Learning Rate Transfercond-mat.dis-nn

The study examines spectral dynamics in deep neural networks, predicting how outliers evolve during training.

problem Understanding spectral evolution in deep neural networks during training.

method Developed a two-level dynamical mean-field theory (DMFT) to track spectral dynamics.

result The theory predicts how outliers evolve with training time, width, output scale, and initialization variance.

spectral dynamics deep neural networks outliers mean-field theory stochastic gradient descent

2026-05-08abs ↗pdf ↗

The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitstat.ML

New Transformer architecture prevents rank degeneracy in deep attention models.

problem Rank degeneracy in deep attention models.

method Modified Softmax-based attention model with skip connections, centered at identity, and scaled logits.

result Existence of a stable SDE implies well-behaved covariance structure, preventing rank degeneracy.

Transformer attention models infinite depth-and-width covariance matrix stochastic differential equation

2023-06-30abs ↗pdf ↗

The Depth-to-Width Interplay in Self-Attentioncs.LG

Self-attention models benefit equally from width and depth, but beyond a certain point, depth becomes less efficient.

problem Understanding the optimal balance between depth and width in self-attention models.

method Theoretical predictions and empirical ablations on networks of varying depths and widths.

result An optimal width of 30K is recommended for a 1-Trillion parameter network, marking a significant width for self-attention models.

self-attention depth width network efficiency parameter size

2020-06-22abs ↗pdf ↗

Finite Versus Infinite Neural Networks: an Empirical Studycs.LG

Empirical study compares wide neural networks to kernel methods, resolving open questions.

problem Understanding the relationship between wide neural networks and kernel methods.

method Large-scale empirical study using various neural network architectures and kernel methods.

result Wide neural networks outperform fully-connected finite-width networks in some cases, but underperform convolutional finite-width networks.

neural networks kernel methods finite width infinite width CIFAR-10

2020-07-31abs ↗pdf ↗

On the linearity of large non-linear models: when and why the tangent kernel is constantcs.LG

Wide neural networks become linear, with constant tangent kernel, due to Hessian scaling.

problem Understanding the linearity of large non-linear models and the tangent kernel.

method Analyzing the scaling properties of the Hessian matrix of neural networks as their width increases.

result The constancy of the tangent kernel is due to the scaling properties of the Hessian matrix.

neural networks tangent kernel Hessian scaling linearity width

2020-10-02abs ↗pdf ↗

Large-width asymptotics for ReLU neural networks with

α

-Stable initializationscs.LG

This paper studies large-width asymptotics for ReLU neural networks with α-Stable initializations.

problem Characterizing the large-width behavior of ReLU neural networks with α-Stable initializations.

method Analysis of the large-width distributions and training dynamics of ReLU neural networks initialized with α-Stable distributions.

result For ReLU neural networks with α-Stable initializations, the large-width training dynamics achieve zero training error at a linear rate, characterized by a random kernel.

ReLU neural networks α-Stable distributions large-width asymptotics training dynamics random kernel

2022-06-16abs ↗pdf ↗

Finite Depth and Width Corrections to the Neural Tangent Kernelcs.LG

We prove the precise scaling, at finite depth and width, for the mean and variance of the neural tangent kernel (NTK) in a randomly initialized ReLU network. The standard deviation is exponential in the ratio of network depth to width. Thus, even in the limit of infinite overparameterization, the NTK is not determinist…

2019-09-13abs ↗pdf ↗

Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Sizecs.LG

New analysis shows optimal embedding learning rate depends on vocabulary size, not just model width.

problem Optimal learning rate for language model embeddings is not well understood, especially with large vocabularies.

method Theoretical analysis of training dynamics, interpolation between

μ

P and LV regimes.

result Optimal embedding learning rate scales as

Θ(\sqrt{width})

in the LV regime, not

Θ(width)

μ

P predicts.

language models learning rate vocabulary size embedding dimension training dynamics

2025-06-17abs ↗pdf ↗

On the Predictability of Pruning Across Scalescs.LG

Pruned neural networks' error scales predictably with architecture and task.

problem Understanding the predictability of pruning across different scales and architectures.

method Functionally approximated the error of pruned networks, showing it is predictable in terms of invariant tying width, depth, and pruning level.

result The error of pruned networks follows a scaling law with interpretable coefficients that depend on architecture and task.

neural networks pruning predictability scaling law architecture

2020-06-18abs ↗pdf ↗

The Geometric Wall: Manifold Structure Predicts Layerwise Sparse Autoencoder Scaling Lawscs.LG

SAEs struggle with curved activation manifolds, revealing layer-dependent scaling laws.

problem Sparse autoencoders' reconstruction error varies across layers, not fitting existing scaling laws.

method Cross-layer study of 844 SAE checkpoints, fitting and regressing on manifold geometry.

result Manifold geometry predicts layer-dependent width exponents in SAEs, with transferable coefficients.

Sparse autoencoders activation manifold layer-dependent scaling geometric mismatch curvature

2026-05-11abs ↗pdf ↗

On the infinite width limit of neural networks with a standard parameterizationcs.LG

There are currently two parameterizations used to derive fixed kernels corresponding to infinite width neural networks, the NTK (Neural Tangent Kernel) parameterization and the naive standard parameterization. However, the extrapolation of both of these parameterizations to infinite width is problematic. The standard p…

2020-01-21abs ↗pdf ↗

Fisher Width: A Geometric Measure of Complexity on Statistical Manifoldscs.LG

Fisher width is a geometric measure of complexity on statistical manifolds.

problem Complexity measures on statistical manifolds

method Introducing Fisher width as a Fisher-geometric analogue of Gaussian width

result Fisher width retains key structural features of Gaussian width while capturing anisotropic geometric effects

Fisher width statistical manifolds complexity measures curvature learning theory

2026-06-16abs ↗pdf ↗

Provable Benefits of Sinusoidal Activation for Modular Additioncs.LG

Sine activation functions enable two-layer neural networks to learn modular addition more efficiently.

problem Learning modular addition with two-layer neural networks.

method Introduced and analyzed sine activation functions, providing theoretical and empirical evidence.

result Sine activation functions allow for constant-width network realizations of modular addition, whereas ReLU networks require linear width scaling.

modular addition two-layer neural networks sine activation expressivity gap generalization bound

2025-11-28abs ↗pdf ↗

High-Dimensional Analysis of Gradient Flow for Extensive-Width Quadratic Neural Networksmath.OC

Study on neural network dynamics in high dimensions with quadratic activation.

problem Understanding training dynamics in overparameterized neural networks.

method Derivation of gradient flow equations and analysis under l2-regularization.

result Characterization of estimator performance and spectral properties in the high-dimensional limit.

neural networks overparameterization quadratic activation gradient flow dynamical mean-field theory

2026-01-15abs ↗pdf ↗

On Random Kernels of Residual Architecturescs.LG

We derive finite width and depth corrections for the Neural Tangent Kernel (NTK) of ResNets and DenseNets. Our analysis reveals that finite size residual architectures are initialized much closer to the "kernel regime" than their vanilla counterparts: while in networks that do not use skip connections, convergence to t…

2020-01-28abs ↗pdf ↗

Quantitative CLTs in Deep Neural Networkscs.LG

Quantitative CLTs show neural network distributions converge to Gaussian as width increases.

problem Understanding the distribution of fully connected neural networks with random weights and biases.

method Analyzing the distribution of a fully connected neural network with random Gaussian weights and biases, proving quantitative bounds on normal approximations.

result The distance between a random fully connected network and the corresponding infinite width Gaussian process scales like

n^{-γ}

for

γ>0

quantitative CLTs deep neural networks random Gaussian weights finite width Gaussian process

2023-07-12abs ↗pdf ↗

Neural Tangents: Fast and Easy Infinite Neural Networks in Pythonstat.ML

Neural Tangents is a library designed to enable research into infinite-width neural networks. It provides a high-level API for specifying complex and hierarchical neural network architectures. These networks can then be trained and evaluated either at finite-width as usual or in their infinite-width limit. Infinite-wid…

2019-12-05abs ↗pdf ↗

Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycs.LG

Recent works have shown that gradient descent can find a global minimum for over-parameterized neural networks where the widths of all the hidden layers scale polynomially with $N$ ( $N$ being the number of training samples). In this paper, we prove that, for deep networks, a single layer of width $N$ following the inpu…

2020-02-18abs ↗pdf ↗

The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimesstat.ML

The study reveals a transition in neural network performance from infinite-width to variance-limited behavior as dataset size increases.

problem Understanding the transition from infinite-width to variance-limited behavior in neural networks.

method Empirical study of the transition from infinite-width to variance-limited behavior as a function of sample size and network width.

result The critical sample size $ P^* $ is approximately $ \sqrt{N} $ for polynomial regression with ReLU networks.

neural networks generalization error finite-width effects variance-limited behavior sample size

2022-12-23abs ↗pdf ↗

Propagation of Chaos in One-hidden-layer Neural Networks beyond Logarithmic Timestat.ML

Study shows polynomial-width neural networks can closely approximate infinite-width networks in polynomial time.

problem Approximating dynamics of polynomial-width neural networks with infinite-width networks.

method Bounding approximation gap through a differential equation governed by mean-field dynamics, considering local Hessian.

result Polynomially many neurons are sufficient to closely approximate mean-field dynamics.

Neural Networks Polynomial Width Infinite Width Mean-Field Dynamics Local Hessian

2025-04-17abs ↗pdf ↗

Linearly Separable Features in Shallow Nonlinear Networks: Width Scales Polynomially with Intrinsic Data Dimensioncs.LG

Shallow nonlinear networks can separate classes linearly with polynomially scaling width.

problem Understanding the linear separability of deep networks' features.

method Modeling inputs as a union of low-dimensional subspaces and using random weights and quadratic activations.

result Shallow nonlinear networks can achieve linear separation with polynomially scaling width.

deep learning nonlinear networks linear separability shallow networks intrinsic dimensionality

2025-01-04abs ↗pdf ↗

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscs.LG

Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available. In this paper, we systematically study model scaling and identify that carefully balancing network depth, width, and resolution can lead to better performan…

2019-05-28abs ↗pdf ↗

Kolmogorov Width Decay and Poor Approximators in Machine Learning: Shallow Neural Networks, Random Feature Models and Neural Tangent Kernelsmath.FA

Poor approximators found in neural networks and random feature models.

problem Understanding why certain neural networks and models perform poorly in approximating functions.

method Established a scale separation of Kolmogorov width type and applied it to neural networks and random feature models.

result Reproducing kernel Hilbert spaces and two-layer neural networks are poor

L^2

-approximators for certain functions.

Kolmogorov width reproducing kernel Hilbert spaces two-layer neural networks random feature models approximation theory

2020-05-21abs ↗pdf ↗

How Does Critical Batch Size Scale in Pre-training?cs.LG

We determine the critical batch size for large language models and find it scales with data size, not model size.

problem Determining the optimal batch size for large-scale model training.

method We propose a measure of critical batch size, pre-trained models, and systematic hyper-parameter sweeps.

result The critical batch size scales primarily with data size, not model size.

critical batch size large language models data parallelism scaling laws hyper-parameters

2024-10-29abs ↗pdf ↗

Finite-Dimensional Gaussian Approximation for Deep Neural Networks: Universality in Random Weightsstat.ML

Study Gaussian approximation for deep neural networks with random weights.

problem Understanding the distribution of deep neural networks with random weights.

method Established Gaussian approximation bounds in Wasserstein-1 norm.

result Convergence rates of order

n^{-({1}/{6})^{L-1} + ε}

for deep networks with proportional layer widths.

Gaussian approximation deep neural networks random weights Wasserstein-1 norm activation function

2025-07-16abs ↗pdf ↗