The paper explores neural scaling laws for deep operator networks, offering a theoretical foundation.
problem Understanding neural scaling laws in deep operator networks.
method Theoretical analysis of approximation and generalization errors.
result Established a theoretical framework to quantify neural scaling laws for deep operator networks.
We discover scaling laws for kernel regression loss under various learning rate schedules.
problem Understanding loss dynamics and learning rate schedules in kernel regression.
method Theoretical analysis of stochastic gradient descent on a power-law kernel regression model.
result Established a Functional Scaling Law (FSL) capturing the full loss trajectory under arbitrary learning rate schedules.
This work analyzes neural scaling laws using power-law data spectra and derives analytical expressions for generalization error.
problem Understanding how neural network performance scales with key factors like data size and model complexity.
method Statistical mechanics techniques applied to one-pass stochastic gradient descent in a student-teacher framework.
result Derivation of analytical expressions for generalization error under power-law data spectra and identification of conditions for power-law scaling.
A simple model explains inference scaling in neural models.
problem Understanding how model performance improves with repeated inference attempts.
method A statistical ansatz based on memorization to study inference scaling laws.
result Inference loss exhibits a power law decay with increasing trials.
Model shows feature learning can improve neural scaling laws for hard tasks.
problem Understanding and improving neural network scaling laws for various task difficulties.
method Developed a solvable model of neural scaling laws, identified three scaling regimes, and demonstrated feature learning's impact on scaling exponents.
result Feature learning can improve scaling with training time and compute for hard tasks, nearly doubling the exponent.
New scaling laws explain deep learning performance growth.
problem Understanding neural network performance growth.
method Analyzed entire training dynamics of various architectures.
result Identified two dynamical scaling laws.
By using numerical simulation, we confirm that Takayasu--Sato--Takayasu (TST) model which leads Pareto's law satisfies the detailed balance under Gibrat's law. In the simulation, we take an exponential tent-shaped function as the growth rate distribution. We also numerically confirm the reflection law equivalent to the…
Scaling laws in linear regression explain model performance improvements with size and data.
problem Disagreement between empirical neural scaling laws and conventional wisdom on variance error.
method Infinite dimensional linear regression setup, one-pass SGD, Gaussian prior, power-law spectrum.
result Variance error is dominated by other errors, disappearing from the bound due to SGD's implicit regularization.
New neural scaling law found for simple quadratic function.
problem Neural scaling laws and their predictions for model performance.
method Analysis of neural networks, lottery ticket ensembling, statistical interpretation.
result Found a new scaling law (α=1) for a simple quadratic function, contradicting previous theories. Study uncovers scaling laws and spectral properties of shallow neural networks.
problem Understanding scaling laws and spectral properties of shallow neural networks.
method Leveraging connections with matrix compressed sensing and LASSO, derived a phase diagram for excess risk.
result Uncovered crossovers between scaling regimes and plateau behaviors, validated empirical observations.
Study how generalization scales with model size and data in quadratic neural networks.
problem Understanding how generalization scales with model size and data in quadratic neural networks.
method Analyzed ℓ2-regularized empirical test error minimization in a quadratic two-layer network with finite-sample setting and structured data. result Revealed a phase diagram with distinct scaling regimes as the number of parameters varies, showing data-dependent power laws controlled by spectral structure of the target.
Scaling laws found for reinforcement learning performance with model size and compute.
problem Challenges in extending generative modeling scaling laws to reinforcement learning.
method Introduced intrinsic performance as a monotonic function of mean episode return.
result Intrinsic performance scales as a power law in model size and environment interactions.
New phases identified in neural scaling laws with compute limits.
problem Understanding neural scaling laws under compute constraints.
method Solved neural scaling model with stochastic gradient descent, derived loss curves, analyzed model-parameter-count phases.
result Identified 4 phases (+3 subphases) in data-complexity/target-complexity phase-plane, derived exponents.
A hierarchical model shows how scaling laws emerge from sequential feature recovery.
problem Emergence of scaling laws from feature learning in multi-layer networks.
method Layer-wise spectral algorithm adapted to compositional structure, sequential feature detection.
result Sequential detection of latent features, leading to explicit power-law decay of prediction error.
SignSGD outperforms SGD in linear regression with optimal scaling laws under PLRF model.
problem Improving linear regression performance with signSGD under power-law random features.
method Analysis of signSGD risk under PLRF model, comparison with SGD, identification of unique effects.
result SignSGD can have a steeper compute-optimal slope than SGD in noisy regimes, especially with WSD schedule.
Predicts language model performance from public models without training.
problem Understanding how language model performance varies with scale.
method Observational approach building scaling laws from publicly available models.
result Smooth, sigmoidal behavior of emergent phenomena is predictable from small models.
Model predicts neural network performance scaling laws across various factors.
problem Understanding the performance of neural networks across different training factors.
method Random feature model trained with gradient descent, analyzing compute-optimal scaling laws.
result Predicts asymmetric compute-optimal scaling rule and behavior of training and test loss gap.
This work explains scaling laws as redundancy laws in deep learning.
problem The mathematical origins of scaling laws in deep learning models remain unclear.
method Kernel regression and analysis of data covariance spectra.
result Scaling laws can be explained as redundancy laws, revealing the learning curve's slope depends on data redundancy.
New mechanism found for power laws including Zipf's law.
problem Understanding the ubiquity of power law distributions.
method Introduced nonlinear self-excited Hawkes processes with fast-accelerating intensities.
result Wide class of nonlinear Hawkes processes have power law intensity PDFs.
This work proves that large models can be compressed significantly without losing performance.
problem Achieving comparable performance with smaller models and less data.
method Developed a universal compression theory for neural networks and datasets.
result Proved that a generic permutation-invariant function can be compressed into a function of polylogarithmic size with vanishing error.
A new scaling law predicts optimal batch size for training models.
problem Finding the optimal batch size for training models efficiently.
method Proposed a three-term scaling law that considers model size, training data, training steps, and batch size.
result The three-term law accurately recovers the optimal batch size and can be robustly fit with fewer training runs.
Large models follow power laws in performance with dataset size or parameters.
problem Understanding neural scaling laws in large language models.
method Joint generative data model and random feature model.
result Modeling and solving the dual limit reveals insights into scaling laws.
This work extends the scaling law to multiple and kernel regression, challenging traditional machine learning principles.
problem Challenging traditional machine learning wisdom with scaling law in large practical models.
method Demonstrates the scaling law in multiple and kernel regression settings.
result The scaling law extends to multiple and kernel regression, providing deeper insights into LLMs.
Study reveals neural scaling laws in random graphs and natural language models.
problem Understanding the origin of neural scaling laws in complex systems.
method Examined scaling laws in transformers trained on random walks and simplified natural language models.
result Neural scaling laws emerge in the absence of power law structure in data correlations.
NN-Turb generates turbulent velocity statistics using neural networks.
problem Creating a 1D field with turbulent velocity statistics.
method Fully-convolutional neural network (NN-Turb) to generate the field.
result NN-Turb generates a 1D field that satisfies Kolmogorov's 2/3 and 4/5 laws, exhibiting intermittency.
Employing profits data of Japanese firms in 2003--2005, we kinematically exhibit the static log-normal distribution in the middle scale region. In the derivation, a Non-Gibrat's law under the detailed balance is adopted together with following two approximations. Firstly, the probability density function of profits gro…
Unified theory for neural scaling laws in hierarchically compositional data.
problem Understanding neural scaling laws in hierarchically compositional data.
method Probabilistic context-free grammars and power-law distributed production rules.
result Unified learning curve behavior for classification and next-token prediction tasks.
Employing profits data of Japanese companies in 2002 and 2003, we confirm that Pareto's law and the Pareto index are derived from the law of detailed balance and Gibrat's law. The last two laws are observed beyond the region where Pareto's law holds. By classifying companies into job categories, we find that companies …
New pruning method breaks power law scaling, potentially reducing error to exponential.
problem Improving neural network performance through scaling alone is costly.
method Developed a new data pruning metric to break power law scaling.
result Pruned datasets show better than power law scaling on various image datasets.
SGD learns two-layer neural networks efficiently, revealing scaling laws.
problem Learning efficiency and scaling laws in SGD for two-layer networks.
method Precise analysis of SGD dynamics for MSE minimization.
result Smooth scaling law in cumulative objective despite abrupt transitions for individual neurons.
We show that an economic system populated by multiple agents generates an equilibrium distribution in the form of multiple scaling laws of conditional PDFs, which are sufficient for characterizing the probability distribution. The existence of the double scaling law is demonstrated empirically for the sales and the lab…
The study explains transformer scaling laws using statistical and approximation theories.
problem Understanding why transformer scaling laws exist for large models trained on low-dimensional data.
method Established statistical estimation and mathematical approximation theories for transformers on low-dimensional manifolds.
result Predicted a power law between generalization error and model and data sizes, with power depending on intrinsic data dimension.
Scaling laws govern predictive uncertainties in deep learning models.
problem Understanding predictive uncertainties in deep learning models.
method Empirical analysis and approximate Bayesian inference on vision and language tasks.
result Scaling laws exist for various measures of predictive uncertainty in deep learning models.
Pruned neural networks' error scales predictably with architecture and task.
problem Understanding the predictability of pruning across different scales and architectures.
method Functionally approximated the error of pruned networks, showing it is predictable in terms of invariant tying width, depth, and pruning level.
result The error of pruned networks follows a scaling law with interpretable coefficients that depend on architecture and task.
Deep learning improves image reconstruction, but scaling up training sets doesn't significantly boost performance.
problem Understanding the impact of training set size on deep learning image reconstruction.
method Empirical study and analytical characterization of performance scaling laws.
result Scaling up training set size does not significantly improve reconstruction quality for deep learning.
Improved scaling laws in linear regression using data reuse.
problem Sustainability of neural scaling laws when running out of new data.
method Data reuse in multi-pass stochastic gradient descent (multi-pass SGD) for M-dimensional linear models trained on N data with sketched features. result Multi-pass SGD achieves a test error of Θ(M1−b+L(1−b)/a) with L>N, improving scaling laws in data-constrained regimes. Study on SGD dynamics and scaling laws for training quadratic neural networks in high dimensions.
problem Optimizing and understanding the training dynamics of quadratic neural networks in high-dimensional settings.
method Sharp analysis of SGD dynamics, combining matrix Riccati differential equations and matrix monotonicity arguments.
result Derivation of scaling laws for prediction risk, highlighting power-law dependencies on optimization time, sample size, and model width.
Power-law portfolios improve diversification by scaling weights sub-linearly.
problem Optimization methods struggle with unstable pair correlations and non-Gaussian risk measures.
method Construct portfolios with penalty proportional to arbitrary order moment of returns, leading to sub-linear weight scaling.
result Infinite order power-law portfolios are perfectly diversified, improving diversification over Kelly portfolios.
Non-Markovian point process shows power-law scaling, similar to nonlinear Markovian process.
problem Understanding the scaling behavior of non-Markovian point processes.
method Analyzed a confined fractional Brownian motion-driven point process and compared it to a nonlinear Markovian process.
result A nonlinear Markovian process can reproduce the power-law scaling behavior of a non-Markovian point process.
Develops a simple model to understand learning curves for arbitrary power laws.
problem Lack of theoretical understanding of scaling laws in machine learning.
method Analyzes a toy model to determine if learning curves are universal or depend on data distribution.
result Determines that learning curves can exhibit n−β for arbitrary power β>0. We study distributions which have both fractal and non-fractal scale regions by introducing a typical scale into a scale invariant system. As one of models in which distributions follow power law in the large scale region and deviate further from the power law in the smaller scale region, we employ 2-dim quantum gravit…
We report the proof that the expression of extended Gibrat's law is unique and the probability distribution function (pdf) is also uniquely derived from the law of detailed balance and the extended Gibrat's law. In the proof, two approximations are employed that the pdf of growth rate is described as tent-shaped expone…
Sloth predicts LLM performance using latent skills across families.
problem Variations in benchmark performance due to differences in training configurations and data processing across model families.
method Sloth uses publicly available benchmark data and assumes LLM performance is driven by latent skills influenced by model size and training tokens. It exploits correlations across benchmarks to provide accurate predictions.
result Sloth predicts LLM performance accurately and offers insights into scaling behaviors for complex tasks.
Sharp theory of neural network scaling laws for hierarchical targets.
problem Learning hierarchical multi-index models in neural networks.
method Sharp information-theoretic scaling laws derived for two-layer neural networks.
result Optimal rates achieved by a simple spectral estimator.
We study the relaxation dynamics of a financial market just after the occurrence of a crash by investigating the number of times the absolute value of an index return is exceeding a given threshold value. We show that the empirical observation of a power law evolution of the number of events exceeding the selected thre…
New findings show neural network training loss follows a power law over time.
problem Understanding the optimization process of neural networks during training.
method Spectral analysis of the integral operator representing the linearized evolution of a large network.
result The loss function in neural network training follows a power law behavior, L(t)∼t−ξ, with exponent ξ determined by network parameters and data characteristics. SAEs struggle with curved activation manifolds, revealing layer-dependent scaling laws.
problem Sparse autoencoders' reconstruction error varies across layers, not fitting existing scaling laws.
method Cross-layer study of 844 SAE checkpoints, fitting and regressing on manifold geometry.
result Manifold geometry predicts layer-dependent width exponents in SAEs, with transferable coefficients.
Analyzed a generalized voter model with power-law herding intensity, revealing anomalous diffusion and long-range memory.
problem Anomalous diffusion and long-range memory in a generalized voter model.
method Derived analytical expressions for moments and first passage time distribution, confirmed numerically.
result The model exhibits long-range memory indicators despite being a Markov model.