Study on how initialization scale affects neural network training regimes.
problem Understanding the transition between kernel and rich regimes in overparametrized models.
method Analysis of simple depth-D models and empirical testing on complex models.
result Scale of initialization controls transition between kernel and rich regimes.
A recent line of work studies overparametrized neural networks in the "kernel regime," i.e. when the network behaves during training as a kernelized linear predictor, and thus training with gradient descent has the effect of finding the minimum RKHS norm solution. This stands in contrast to other studies which demonstr…
Exact solutions reveal how unbalanced initializations promote rapid feature learning in neural networks.
problem Understanding how neural networks efficiently extract features from data.
method Deriving exact solutions to a minimal model of neural networks transitioning between lazy and rich learning regimes.
result Unbalanced layer-specific initialization variances and learning rates determine the degree of feature learning.
Study on rich regime training in deep learning, finding active parameters in bottom layers.
problem Understanding the practical success of deep learning models.
method Empirical study on rich regime training with benchmark datasets, re-initialization analysis, and probabilistic Layer-Wise Sparse SGD.
result Probabilistic Layer-Wise Sparse SGD matches vanilla SGD's generalization performance with improved efficiency.
The paper explains the richness scale of wide neural networks.
problem Understanding the behavior of overparameterized neural networks.
method Nonrigorous derivation and empirical evidence.
result Wide neural networks exhibit a richness scale from lazy kernel behavior to feature learning.
Motivated by the widespread adoption of large-scale A/B testing in industry, we propose a new experimentation framework for the setting where potential experiments are abundant (i.e., many hypotheses are available to test), and observations are costly; we refer to this as the experiment-rich regime. Such scenarios requ…
New metric measures dynamical richness without relying on accuracy.
problem Lack of a reliable metric for measuring dynamical richness.
method Developed a computationally efficient, performance-independent metric based on low-rank bias.
result Metric recovers neural collapse as a special case and captures known transitions without accuracy.
Use simplified layerwise linear models to understand neural dynamics.
problem Complex neural network dynamics are hard to grasp.
method Apply simplified layerwise linear models to explain neural phenomena.
result Simplified models explain neural collapse, emergence, etc.
A large number of online services provide automated recommendations to help users to navigate through a large collection of items. New items (products, videos, songs, advertisements) are suggested on the basis of the user's past history and --when available-- her demographic profile. Recommendations have to satisfy the…
Study shows optimal model performance at critical level of feature learning.
problem Catastrophic forgetting in neural networks, especially in non-stationary environments.
method Systematic study on model scale and feature learning, using dynamical mean field theory.
result Optimal performance achieved at a critical level of feature learning, dependent on task non-stationarity and model scale.
Transformers capture combinatorial tasks with bounded error and logarithmic sample dependence.
problem Capturing complex combinatorial tasks with bounded error and sample efficiency.
method Formal definition of algorithmic capture, empirical analysis of infinite-width transformers, upper bounds on computational complexity.
result Transformers exhibit an inductive bias favoring simpler algorithmic procedures over higher complexity ones.
Optimal treatment regime uses proxy variables to improve decision-making.
problem Insufficient covariates in observational data lead to confounding issues.
method Proximal causal inference framework and outcome/treatment confounding bridges.
result The proposed optimal treatment regime outperforms existing ones.
Grokking occurs when neural networks transition from lazy to rich training dynamics, fitting initial features before generalizing.
problem Understanding why neural networks exhibit early train loss decrease without corresponding test loss improvement.
method Analyzing vanilla gradient descent on polynomial regression with a two-layer neural network, identifying sufficient statistics for test loss.
result Grokking arises when a network first attempts to fit a kernel regression solution with initial features, followed by late-time feature learning.
Analysis of deep neural networks under various learning rules reveals dynamics of feature and prediction learning.
problem Understanding how different learning rules affect feature and prediction dynamics in deep neural networks.
method Analysis of infinite-width deep networks trained with gradient descent and various learning rules.
result The evolution of the output function is governed by an effective neural tangent kernel (eNTK), which varies depending on the learning rule and training regime.
New theory predicts deep neural networks can operate in an extended critical regime without fine-tuning.
problem Understanding the dynamics and computational principles of deep neural networks.
method Combining theories of heavy-tailed random matrices and non-equilibrium statistical physics.
result Deep neural networks can operate in an extended critical regime without fine-tuning parameters.
Optimal dividends strategy in a two-state regime-switching environment.
problem Maximizing profits from dividends until bankruptcy in a company with fluctuating cash surplus and regime changes in drift, volatility, and bankruptcy levels.
method Analyzes the optimal dividend payout strategy considering four factors: Brownian fluctuations in cash surplus, regime changes in drift, volatility, and bankruptcy levels.
result Rich structure of the optimal strategy, which can be either barrier-type or liquidation-barrier type, depending on model parameters.
New bounds for high-dimensional sparse linear bandits, balancing information and regret.
problem Stochastic linear bandits with high-dimensional sparse features.
method Derivation of minimax regret lower and upper bounds for explore-then-commit algorithm.
result Optimal rate of Θ ( n 2 / 3 ) Θ(n^{2/3}) Θ ( n 2/3 ) for data-poor regime, complemented by O ( n ) O(\sqrt{n}) O ( n ) under signal magnitude assumption. This study reveals efficient finite-difference computation for gradient regularization in deep learning.
problem Improving generalization performance in deep learning through gradient regularization.
method Analyzes and reveals a specific finite-difference computation that reduces computational cost and improves generalization performance.
result Finite-difference computation strengthens the implicit bias towards rich regimes and enhances generalization performance.
Develops a new framework to analyze gradient flow regimes and derive explicit solutions.
problem Analyzing scaling regimes and deriving explicit analytic solutions for gradient flow in large learning problems.
method Formal power series expansion of the loss evolution with coefficients encoded by diagrams.
result Reveals different learning phases and obtains explicit solutions in some cases.
The study analyzes transfer learning in infinite-width neural networks, improving generalization on target tasks.
problem Improving generalization in neural networks when using pretraining on a source task.
method Developed a theory under gradient flow for infinitely wide networks, analyzing fine-tuning and joint pretraining.
result Summary statistics of randomly initialized networks after pretraining are adaptive kernels that depend on both source and target data.
Study improves predictive models for ICU data across hospitals.
problem Degradation of predictive model performance in new hospitals.
method Anchor regression and anchor boosting for domain generalization.
result Anchor regularization enhances out-of-distribution performance.
Motivated by the study of Fano type varieties we define a new class of log pairs that we call asymptotically log Fano varieties and strongly asymptotically log Fano varieties. We study their properties in dimension two under an additional assumption of log smoothness, and give a complete classification of two dimension…
When can reliable inference be drawn in the "Big Data" context? This paper presents a framework for answering this fundamental question in the context of correlation mining, with implications for general large scale inference. In large scale data applications like genomics, connectomics, and eco-informatics the dataset…
Optimization of neural networks scales with γ, revealing unique loss curves and optimal learning rates.
problem Understanding the impact of feature learning strength on neural network optimization.
method Empirical investigation of neural networks with varying γ, analyzing the γ γ γ - η η η plane, and examining loss curves. result Optimal learning rate scales non-trivially with γ, with η ∗ ∝ γ 2 η^* \propto γ^2 η ∗ ∝ γ 2 for small γ and η ∗ ∝ γ 2 / L η^* \propto γ^{2/L} η ∗ ∝ γ 2/ L for large γ. New model predicts energy prices under different scenarios.
problem Complex causal relationships in energy markets with continuous regime changes.
method Augmented Time Series Structural Causal Models (ATSCM) integrating neural causal discovery.
result Enables novel counterfactual queries in energy markets.
Housing markets play a crucial role in economies and the collapse of a real-estate bubble usually destabilizes the financial system and causes economic recessions. We investigate the systemic risk and spatiotemporal dynamics of the US housing market (1975-2011) at the state level based on the Random Matrix Theory (RMT)…
Simplifies deep learning scaling analysis without sacrificing accuracy.
problem Interpreting feature learning mechanisms and determining network implicit bias in high-dimensional settings.
method Developed a heuristic approach for predicting data and width scales of feature learning patterns.
result Predictions align with known results and extend to complex architectures.
The study provides precise asymptotic theory for in-context learning by Transformers.
problem Understanding the sample complexity, pretraining task diversity, and context length for successful in-context learning.
method An exactly solvable model of linear regression task by linear attention, deriving sharp asymptotics.
result Double-descent learning curve with increasing pretraining examples, phase transition between low and high task diversity regimes.
New findings support a new community recovery threshold for Stochastic Block Model with many communities.
problem Recovering communities in Stochastic Block Model with more than sqrt(n) communities.
method Counting specific motifs to achieve polynomial-time community recovery above a new threshold.
result LDP fails below the new threshold, but polynomial-time recovery is possible above it.
Adaptive kernels from neural networks improve model performance.
problem Improving neural network performance through adaptive kernels.
method Deriving adaptive kernels from infinite-width neural networks using feature learning and gradient flow training.
result Adaptive kernels achieve lower test loss compared to traditional kernels.
Improved generative models using overparametrized shallow neural networks.
problem Improving generative models for data with hidden low-dimensional structure.
method Using energy-based models with overparametrized shallow neural networks as approximators.
result Models trained in the 'active' regime outperform those in the 'lazy' or kernel regime, leading to better adaptivity to hidden structure.
Statistical physics explains deep learning's feature learning capacity.
problem Understanding neural networks' ability to learn complex features.
method Study of a multi-layer perceptron in the interpolation regime.
result Optimal learning requires specialization across layers and neurons.
Many studies in biomedical and health sciences involve small sample sizes due to logistic or financial constraints. Often, identifying weak (but scientifically interesting) associations between a set of predictors and a response necessitates pooling datasets from multiple diverse labs or groups. While there is a rich l…
New model controls memory in seq2seq tasks, revealing learning regimes.
problem Understanding memory in seq2seq tasks using neural networks.
method Introducing a stochastic switching-Ornstein-Uhlenbeck (SSOU) model to control memory and a measure of non-Markovianity.
result Two learning regimes emerge from the interplay of time scales in the SSOU process.
New method estimates chirp parameters robustly from noisy mixtures.
problem Estimating chirp parameters from noisy mixtures of higher-order polynomials.
method Modified Langevin Monte Carlo (LMC) with curvature guidance.
result CG-LMC algorithm reliably finds minimizer in low SNR regimes.
Quantum Reservoir Computing classifies complex probability distributions and identifies volatility regimes.
problem Statistical and financial classification problems with heavy-tailed distributions and correlated time series.
method Implemented QRC in a superconducting quantum circuit with Josephson junctions.
result QRC outperforms classical methods in limited information scenarios.
Proposes a framework to balance supervised and unsupervised learning using random matrix theory.
problem Balancing supervised and unsupervised learning in high-dimensional data.
method QLDS model with quadratic margin maximization under low density separation assumption.
result Establishes a smooth bridge between supervised and unsupervised learning methods.
CoreFlow models matrix-valued distributions efficiently, preserving shared low-rank structure.
problem Learning matrix-valued distributions from high-dimensional and incomplete data.
method Low-rank flow model that learns shared row/column subspaces and trains a normalizing flow on the core.
result CoreFlow improves generation quality in few-sample regimes and remains competitive in data-rich settings.
Neural networks can learn kernel machines with a data-dependent kernel.
problem Can neural networks in the rich feature learning regime learn a kernel machine?
method Demonstrated silent alignment effect in neural networks, showing they can learn a kernel machine with a data-dependent kernel.
result Neural networks in the rich feature learning regime can learn a kernel machine with a data-dependent kernel due to silent alignment.
The study reveals a transition in neural network performance from infinite-width to variance-limited behavior as dataset size increases.
problem Understanding the transition from infinite-width to variance-limited behavior in neural networks.
method Empirical study of the transition from infinite-width to variance-limited behavior as a function of sample size and network width.
result The critical sample size \( P^* \) is approximately \( \sqrt{N} \) for polynomial regression with ReLU networks.
Lower bounds on eigenspectrum show rich action spaces force polynomial regret in linear bandits.
problem Understanding the minimum eigenvalue growth in linear bandits with rich action sets.
method Non-asymptotic lower bound on eigenspectrum of design matrix.
result Minimum eigenvalue of expected design matrix grows as Ω ( n ) Ω(\sqrt{n}) Ω ( n ) for sub-linear regret. Theory captures feature learning effects in finite CNNs.
problem Feature learning in finite deep neural networks.
method Derive a self-consistent Gaussian Process theory.
result Good agreement with experiments and sharp transition between regimes.
This paper extends RMT for deep learning models beyond eigenvalues.
problem Challenges in high-dimensional, overparameterized ML models.
method Introduces High-dimensional Equivalent to analyze nonlinear models.
result Unified understanding of training and generalization in deep learning.
Ridge regression reveals surprising high-dimensional behaviors via random matrix theory.
problem Understanding power-law scalings in high-dimensional regression models.
method Random matrix theory and free probability.
result Analytic formulas for training and generalization errors derived from S S S -transform. Neural networks learn task-specific features, influenced by nonlinearity.
problem Understanding the nature of task-dependent feature learning in neural networks.
method Investigation of fully-connected, wide neural networks using Bayesian framework.
result The nature of internal representations depends on neuronal nonlinearity, leading to analog, redundant, or sparse coding schemes.
Study reveals how initialization scale affects training accuracy in linear networks.
problem Understanding implicit bias in linear classification models.
method Asymptotic analysis of gradient flow trajectories and training loss minimization.
result Implicit bias is more complex at reasonable initialization scales and training accuracies.
IDS optimizes regret in stochastic partial monitoring with linear rewards.
problem Optimizing decision-making in uncertain environments with linear rewards.
method Information Directed Sampling (IDS) for stochastic partial monitoring.
result Achieves optimal regret rates in all observable game regimes.
Develops methods to simulate rare transitions in molecular systems.
problem Rare transitions between metastable states in molecular systems are difficult to study due to limited data.
method Two novel methods: chain-based and midpoint-based approaches.
result Demonstrates effectiveness of methods in both data-rich and data-scarce scenarios.