Grokking occurs at numerical stability edge, requiring regularization to prevent.
problem Delayed generalization in deep learning models.
method Identified Softmax Collapse (SC) as the cause of grokking without regularization.
result Mitigating SC enables grokking without regularization.
Analyzes why neural networks generalize beyond training data.
problem Understanding why neural networks generalize beyond training data.
method Examined loss landscapes of neural networks to identify the mismatch between training and test losses.
result Identified the 'LU mechanism' explaining grokking in various tasks.
We prove grokking in ridge regression, showing overfitting doesn't guarantee good generalization.
problem The onset of generalization long after overfitting in over-parameterized linear regression models.
method Proved end-to-end grokking results for learning over-parameterized linear regression models using gradient descent with weight decay.
result Generalization error eventually becomes arbitrarily small, but poor generalization persists long after overfitting.
Novel anti-grokking phase discovered in neural networks, revealed by HTSR layer quality metric.
problem Understanding and detecting overfitting in neural networks.
method 3-layer MLP, weight decay, HTSR layer quality metric α, correlation traps, Kolmogorov–Smirnov tests. result Anti-grokking phase detected late in training, revealed by α<2 and correlation traps. The paper explores grokking in various models, revealing it's not limited to neural networks.
problem Understanding grokking in different models and its mechanism.
method Empirical exploration of grokking in neural networks, Gaussian processes, linear regression, and Bayesian neural networks. Inducing grokking through spurious dimensions.
result Grokking occurs in non-neural architectures, indicating it's not model-specific.
Regularization can induce grokking in neural networks, improving generalization.
problem Delayed generalization following overfitting in neural networks.
method Demonstrates that gradient descent with small regularization of model properties induces grokking.
result Regularization can induce grokking, extending previous work on weight decay.
Study explores grokking in neural networks, revealing transition from memorization to generalization.
problem Understanding the transition from memorization to generalization in over-parameterized neural networks.
method Extensive experiments and exploration of various viewpoints on grokking mechanism.
result Sharp transition from no generalization to perfect generalization observed during prolonged training.
Linear models can grok without understanding, improving generalization.
problem Understanding the phenomenon of grokking in linear models.
method Analytical and numerical derivation of training and generalization dynamics in linear networks.
result Grokking can occur in linear networks without reaching understanding, and its timing depends on various parameters.
Deep neural networks can grok better than shallow ones, showing multi-stage generalization.
problem Understanding the generalization behavior of deep neural networks.
method Empirical replication and analysis of grokking phenomenon in deep MLP models.
result Deep neural networks exhibit multi-stage generalization, with a secondary surge in test accuracy.
Deep networks learn features suddenly, akin to a phase transition.
problem Understanding sudden feature learning in deep networks.
method Adaptive kernel approach applied to two teacher-student models.
result Feature learning and Grokking are analogous to a first-order phase transition.
Grokking occurs in simple binary logistic classification near linear separability and noise.
problem Delayed generalization in binary logistic classification.
method Analytical and empirical insights into gradient descent dynamics near critical points.
result Logistic regression exhibits grokking when training data is nearly linearly separable from the origin with strong noise.
The paper explains how neural networks learn less salient frequency components during training.
problem Understanding the grokking phenomenon in neural networks.
method Empirical frequency analysis of training data.
result Neural networks initially learn less salient frequency components of the test data.
GrokAlign aligns Jacobians to accelerate grokking in deep networks.
problem Accelerating the training dynamics of deep networks to avoid delayed generalisation and robustness.
method Aligning the Jacobians of a deep network with the training data to ensure grokking under a low-rank assumption.
result GrokAlign regularizes Jacobians to induce grokking sooner than conventional methods.
Theoretical study explains grokking in neural networks.
problem Understanding the abrupt transition from fitting to generalizing in neural networks.
method Characterized a shell-core topological configuration of the solution space induced by Adam's optimization dynamics.
result Derived grokking scaling laws for learning rate, batch size, and regularization coefficient.
SLT reveals how grokking occurs via basin selection in training.
problem Understanding grokking in machine learning models.
method Singular Learning Theory (SLT) to analyze the loss landscape and local learning coefficient (LLC).
result LLC ranks basins by statistical preference, leading to grokking.
Grokking occurs when neural networks transition from lazy to rich training dynamics, fitting initial features before generalizing.
problem Understanding why neural networks exhibit early train loss decrease without corresponding test loss improvement.
method Analyzing vanilla gradient descent on polynomial regression with a two-layer neural network, identifying sufficient statistics for test loss.
result Grokking arises when a network first attempts to fit a kernel regression solution with initial features, followed by late-time feature learning.
Persistent homology reveals a topological signature of grokking in neural networks.
problem Understanding how neural networks learn and generalize from modular arithmetic tasks.
method Persistent homology on point clouds derived from embedding matrices of models trained on modular arithmetic.
result A sharp increase in first homology persistence indicates grokking, with a dominant long-lived topological feature and structured secondary features.
The paper introduces a diagnostic method to detect grokking transitions in models before test accuracy improves.
problem Detecting the transition from training to generalization in machine learning models.
method Summarize task-dependent observables as empirical distributions, map them to Wasserstein/quantile coordinates, and analyze using Hankel dynamic mode decomposition.
result The diagnostic method achieves AUROC \(\approx\) 0.93 for grokking-vs-non-grokking discrimination at the run level.
New measure LMN explains neural network grokking.
problem Delayed generalization after memorization in neural networks.
method Defined LMN to measure network complexity, showing LMN correlates with test losses linearly.
result LMN reveals intriguing XOR network behavior and is a promising complexity measure.
Theoretical analysis explains why models generalize after overfitting in modular addition.
problem Understanding why models generalize after overfitting in modular addition.
method Theoretical analysis and gradient descent behavior of two-layer quadratic networks and Transformers.
result Two-layer quadratic networks and simple Transformers generalize well after initially overfitting, indicating grokking.
Holomorphic networks on modular arithmetic show clear success or failure, no in-between.
problem Understanding when neural networks can represent modular arithmetic tasks.
method Two-layer networks with holomorphic monomial activations trained on modular tasks.
result The network's output is confined to a subspace of characters, and representability depends on the task's Fourier support.
Neural networks can overfit perfectly to noisy data and then grok near-optimal generalization.
problem Neural networks' ability to overfit perfectly to noisy data and then generalize near-optimally.
method Two-layer ReLU networks trained by gradient descent on XOR cluster data.
result Neural networks can achieve perfect fit to noisy training data and then grok near-optimal generalization.
Gradient flow with weight decay shows grokking effect in deep learning.
problem Understanding the grokking effect in deep learning.
method Analyzing gradient flow dynamics with weight decay.
result Weight decay causes slow norm reduction, explaining grokking.
Paper proposes GrokTransfer to eliminate delayed generalization in neural networks.
problem Delayed generalization in neural networks, compromising predictability and efficiency.
method Trains a smaller, weaker model to reach a nontrivial test performance, then uses its learned input embedding to initialize the stronger model.
result GrokTransfer enables the target model to generalize directly without delay, across various tasks.
A simple model explains deep learning phenomena like grokking and gradient boosting.
problem Understanding the unexpected behaviors of deep learning models.
method A telescoping sequence of first-order approximations to explain neural network performance.
result Empirical insights into neural network performance and training process.
Bayesian analysis reveals epistemic uncertainty as a key diagnostic for delayed generalization in in-context learning.
problem Delayed generalization in in-context learning from few examples.
method Bayesian perspective, modular arithmetic tasks, approximate Bayesian techniques, spectral mechanism analysis.
result Epistemic uncertainty collapses sharply when the model groks, indicating a practical diagnostic of generalization.
Neural networks learn modular arithmetic but not all, extending known solutions to generalize.
problem Neural networks struggle with modular arithmetic, especially for polynomials.
method Developed analytical solutions for MLP networks to learn modular addition and multiplication, then combined these solutions to generalize on arbitrary modular polynomials.
result Neural networks can learn and generalize solutions to modular polynomials, supporting the hypothesis that some polynomials are learnable.
Recursive Feature Machines show grokking in modular arithmetic without neural networks.
problem Grokking in modular arithmetic tasks.
method Recursive Feature Machines (RFM) with Average Gradient Outer Product (AGOP).
result RFM and neural networks learn block-circulant features to solve modular arithmetic.
Breaking symmetry in training data is key for generalization in feature learning kernels.
problem Grokking in algebraic tasks, where models perform well on training but fail on unseen data.
method Used Recursive Feature Machine (RFM) with AGOP to learn task-relevant features, breaking symmetry in training data.
result Generalization occurs only when symmetry in the training set is broken, and RFM generalizes by recovering underlying invariance group action.
Use simplified layerwise linear models to understand neural dynamics.
problem Complex neural network dynamics are hard to grasp.
method Apply simplified layerwise linear models to explain neural phenomena.
result Simplified models explain neural collapse, emergence, etc.
We explain how neural networks learn to solve modular addition tasks.
problem How two-layer neural networks learn to solve modular addition tasks.
method Formalized a diversification condition during training, proving it allows the network to approximate the correct logic for modular addition.
result Neural networks can robustly identify the correct sum through phase symmetry and frequency diversification.
Study interprets neural network generalization and memorization on corrupted data.
problem Understanding when a neural network has memorized corrupted data versus learned the underlying rule.
method Analyzes multi-layer perceptrons and Transformers on modular arithmetic tasks with corrupted labels.
result Regularization methods can force networks to ignore corrupted data, improving accuracy on uncorrupted data.
SLT explains neural network success by closing theory-practice gap.
problem Failure of classical inference and learning theory in modern neural networks.
method Physics-inspired Singular Learning Theory (SLT) applied to neural networks.
result SLT recovers known and novel scaling laws for neural network phase transitions.
Derives FACT, an alternative to NFA for neural networks, explaining feature learning.
problem Understanding how neural networks learn representations.
method First-principles approach using first-order optimality conditions.
result FACT explains why NFA holds and provides a principled alternative.
Theory explains generalization in deep learning, reducing memorization and improving performance.
problem Understanding and improving generalization in deep learning models.
method Developed a non-asymptotic theory using the empirical neural tangent kernel.
result Generalization is possible even when the kernel evolves significantly, with coherent signal accumulation and noise suppression.
RNNs solve modular addition tasks using low rank and sparse Fourier structures.
problem Solving modular addition tasks with recurrent neural networks.
method Identified low rank structures and sparse Fourier representations in RNN weights.
result RNNs robust to removing individual frequencies but degrade with more ablation.
New metric measures dynamical richness without relying on accuracy.
problem Lack of a reliable metric for measuring dynamical richness.
method Developed a computationally efficient, performance-independent metric based on low-rank bias.
result Metric recovers neural collapse as a special case and captures known transitions without accuracy.
Ordering examples in modular arithmetic training affects model performance.
problem Training efficiency and model performance in modular arithmetic.
method Varying example ordering while keeping other conditions constant.
result Adversarially structured ordering suppresses learning, while fixed-ordering strategies achieve high accuracy.
New theory explains how chaotic training improves neural network generalization.
problem Understanding how chaotic training improves neural network generalization.
method Representing stochastic optimizers as random dynamical systems and introducing a new dimension concept.
result Generalization in chaotic training depends on the complete Hessian spectrum and partial determinants.
Logit regularization induces logit clustering, affecting classifier performance.
problem Understanding the mechanism of logit regularization in classification.
method Analysis of logit regularization in linear classification, proving logit clustering leads to Fisher's Linear Discriminant alignment.
result Logit regularization can halve critical sample complexity and induce robust generalization.
Reasoning models outperform LLMs on CFA exams.
problem Previous research showed LLMs failing CFA exams; reasoning models show promise.
method Evaluated state-of-the-art reasoning models on CFA exams using pass/fail criteria.
result Most reasoning models pass all three CFA levels; Gemini 3.0 Pro achieves highest scores.
Transformers are less sensitive to input perturbations compared to other models.
problem Understanding the inductive biases of transformers and distinguishing them from other architectures.
method Identified token-wise sensitivity as a metric to explain transformers' inductive biases across different data modalities.
result Transformers have lower sensitivity than MLPs, CNNs, ConvMixers, and LSTMs, across vision and language tasks.
Noise stability improves understanding of Transformer models.
problem Lack of robustness metrics for real-valued domains and junta-like input dependence in modern LLMs.
method Proposed noise stability as a new metric and developed a practical regularization method.
result Noise stability regularization method accelerates training by 35-75%.
Classifier detects LLM-generated text with guarantees.
problem Detecting fake LLM-generated text to prevent misuse.
method Trained classifier without auxiliary info, distinguishing human and LLM text.
result Achieves higher accuracy than existing detectors with type-I error control.
New optimization method helps models generalize better after achieving near-perfect training performance.
problem Models can achieve near-perfect training performance but fail to generalize well to unseen examples.
method GROKtimizer combines rapid convergence to interpolation with post-interpolation norm minimization using Critically Damped Momentum.
result GROKtimizer provides a quadratic speedup over classical gradient descent, offering a natural solution for selecting low-norm interpolating solutions.
RLVR training dynamics reveal an implicit curriculum that shapes learning progression.
problem Understanding how RLVR overcomes the long-horizon barrier.
method Developed a theory of training dynamics for RLVR on transformers, using Fourier analysis on finite groups.
result Mixed-difficulty training naturally follows an implicit curriculum, shaping the learning progression from easy to hard.
RLVR learning dynamics naturally create an implicit curriculum for transformers.
problem How rewards based on final outcomes help overcome the long-horizon barrier in reasoning models.
method Developed a theory of training dynamics for RLVR on transformers, using Fourier analysis on finite groups.
result Mixed-difficulty training induces an implicit curriculum that shapes the learning progression from easy to hard.
Framework analyzes neural network dynamics for better understanding and optimization.
problem Understanding the fundamental mechanisms of deep neural networks.
method Dynamical systems theory, transformation units, attraction basins.
result Different transformation modes lead to distinct learning phases and network performance.