OUI tool detects optimal Weight Decay for DNNs without validation data.
problem Optimal Weight Decay hyperparameter selection for DNNs.
method Overfitting-Underfitting Indicator (OUI) tool.
result OUI correlates with improved generalization and validation scores.
This paper investigates the effectiveness of decoupled weight decay at the start of training.
problem The traditional approach to weight decay is not effective throughout training.
method The authors investigate decoupled weight decay, applying it only at the start of training.
result Applying weight decay only at the start of training stabilizes network weights and improves performance.
TDA detects financial bubbles through early warning signals.
problem Detecting financial bubbles early.
method Using Log-Periodic Power Law Singularity (LPPLS) model to fit financial time series data.
result TDA generates early warning signals when LPPLS model fits the data.
Variational Laplace improves Bayesian neural network performance without sampling.
problem Improving Bayesian neural network performance and calibration.
method Develops a new variational Laplace method for BNNs, exploiting curvature of likelihood.
result Variational Laplace outperforms standard VI methods in test performance and calibration.
A new distillation method transfers channel information from teacher to student.
problem Transfer knowledge from teacher to student with fewer parameters and calculations.
method Channel Distillation (CD) and Guided Knowledge Distillation (GKD) with loss decay.
result Achieved 27.68% top-1 error on ImageNet with ResNet18, outperforming state-of-the-art methods.
New insights into how to inspect and learn from multi-stage processes and AI reasoning.
problem Understanding how to attribute outcomes to early stages in multi-stage operations and AI reasoning.
method Information-theoretic analysis and mathematical proofs of four key results.
result Uniform checkpoint spacing is minimax-optimal for inspection design under homogeneous signal attenuation.
Theoretical framework explains why few epochs are enough for LLM fine-tuning.
problem Understanding why few epochs are sufficient for LLM fine-tuning.
method Combining early stopping theory with attention-based Neural Tangent Kernel (NTK) for LLMs.
result Formalizes convergence rate of attention-based fine-tuning with respect to sample size.
Hybrid regularization avoids double descent in random feature models.
problem Avoiding the double descent phenomenon in random feature models.
method Combines early stopping and weight decay, using GCV for hyperparameter selection.
result Hybrid method successfully avoids double descent and achieves comparable generalization.
We derive explicit formulas for time decay, for the European call and put options at expiry, and use them to calculate analytical approximations to the price of the American put and early exercise boundary near expiry. We show that for many families of non-Gaussian processes used in empirical studies of financial marke…
SAD-DPSGD improves model performance on imbalanced medical datasets like HAM10000.
problem Data leakage and imbalanced distribution in medical image classification datasets.
method SAD-DPSGD uses a linear decaying mechanism for noise and clipping thresholds to enhance performance.
result SAD-DPSGD outperforms Auto-DPSGD on HAM10000, improving accuracy by 2.15%.
Variational Laplace improves Bayesian neural networks performance.
problem Improving Bayesian neural networks performance.
method Develops variational Laplace for BNNs, exploiting curvature of likelihood.
result Variational Laplace outperforms other inference methods.
HED Score improves temporal evaluation of detection accuracy.
problem Temporal agnosticism in existing evaluation frameworks for non-stationary processes.
method Measure-theoretic HED Score integrating exponentially decaying kernel over posterior probability stream.
result HED Score achieves 388.8% improvement over ROC/AUC on NSL-KDD benchmark.
We introduce a new weight-decay scaling rule to maintain sublayer gains across different widths in modern scale-invariant architectures.
problem In modern scale-invariant architectures, training quickly enters a steady state where normalization layers create backward scale sensitivity, degrading learning-rate transfer.
method We introduce a weight-decay scaling rule for AdamW that preserves sublayer gain across widths by equalizing the effective learning rate.
result Our empirical weight-decay scaling rule λ 2 ∝ d λ_2\propto \sqrt{d} λ 2 ∝ d approximately keeps sublayer gains width invariant, enabling zero-shot transfer of learning rate and weight decay. Adversarial training effectiveness varies widely due to inconsistent training settings.
problem Variability in adversarial training effectiveness due to inconsistent training settings.
method Comprehensive evaluation of 10+ adversarial training methods and their hyperparameters.
result Basic training settings like weight decay can significantly impact adversarial robustness.
Quantum systems with scrambling improve temporal information processing, but scaling requires exponential overhead.
problem Scalability and memory retention of quantum reservoirs in temporal information processing.
method Examined a quantum reservoir processing framework with scrambling reservoirs modeled by high-order unitary designs, analyzed in noiseless and noisy settings.
result Memory retention improves exponentially with reservoir size but worsens with reservoir iterations, requiring exponential shot overhead for scaling.
Gradient descent trains neural networks to match kernel regression's sharp generalization rate.
problem Training over-parameterized neural networks for nonparametric regression.
method Gradient descent with early stopping on over-parameterized two-layer neural networks.
result Trained neural networks achieve sharp generalization rate of O ( ε n 2 ) \mathcal{O}(ε_n^2) O ( ε n 2 ) . Regularization is typically understood as improving generalization by altering the landscape of local extrema to which the model eventually converges. Deep neural networks (DNNs), however, challenge this view: We show that removing regularization after an initial transient period has little effect on generalization, ev…
We prove grokking in ridge regression, showing overfitting doesn't guarantee good generalization.
problem The onset of generalization long after overfitting in over-parameterized linear regression models.
method Proved end-to-end grokking results for learning over-parameterized linear regression models using gradient descent with weight decay.
result Generalization error eventually becomes arbitrarily small, but poor generalization persists long after overfitting.
SA-PEF improves federated learning efficiency by correcting gradient mismatches.
problem Slow decay of residual error in federated learning under non-IID data.
method Integrates step-ahead correction with partial error feedback.
result Achieves faster convergence and target accuracy compared to standard EF.
Self-training in linear models shows a U-shaped test-risk curve due to signal forgetting and denoising.
problem Understanding the dynamics of iterative self-training in high-dimensional linear regression.
method Derivation of deterministic-equivalent recursions for prediction risk and effective noise, analysis of signal forgetting and denoising effects.
result An optimal early-stopping time is determined, and a U-shaped test-risk curve is observed.
The study analyzes how many neurons are needed for two-layer neural networks trained with gradient descent.
problem Determining the minimum number of neurons required for effective training of shallow neural networks.
method Analyzes two-layer neural networks in the NTK regime, trained with gradient descent. Derives fast rates of convergence and tracks the number of hidden neurons required for generalization.
result Derives fast rates of convergence and improves on existing results for the number of hidden neurons needed for generalization.
Extends Minkowski stability proof to minimal decay assumptions.
problem Global stability of Minkowski spacetime with minimal decay.
method Extends Christodoulou-Klainerman's proof to minimal decay assumptions.
result Exterior stability of Minkowski holds with borderline decay.
Empirical study compares wide neural networks to kernel methods, resolving open questions.
problem Understanding the relationship between wide neural networks and kernel methods.
method Large-scale empirical study using various neural network architectures and kernel methods.
result Wide neural networks outperform fully-connected finite-width networks in some cases, but underperform convolutional finite-width networks.
New method creates vacuum data at minimal and borderline decay thresholds.
problem Creating vacuum initial data at specific decay thresholds.
method Conical solution-operator method applied to vacuum asymptotically flat initial data.
result Demonstrates global and exterior stability of Minkowski spacetime.
We extend the Deep Image Prior (DIP) framework to one-dimensional signals. DIP is using a randomly initialized convolutional neural network (CNN) to solve linear inverse problems by optimizing over weights to fit the observed measurements. Our main finding is that properly tuned one-dimensional convolutional architectu…
Unique solutions found for wave-like decaying null infinity equations.
problem Wave-like decaying null infinity equations with spherically symmetric Einstein-scalar-field.
method Local and global unique solutions for small initial data.
result Sharp decaying condition for unique solutions.
Early stopping improves logistic regression's calibration and consistency in high dimensions.
problem Improving the statistical performance of gradient descent in overparameterized logistic regression.
method Investigates the effects of early stopping on gradient descent in logistic regression.
result Early-stopped gradient descent is well-calibrated and statistically consistent, while asymptotic gradient descent is not.
New findings on flatness of certain metrics with fast decay.
problem Rigidity of positive mass theorem under fast metric decay.
method Considered metrics with nonnegative scalar curvature and rapid decay at infinity.
result Any such metric is necessarily flat in dimensions 4 and higher if decay rate exceeds Schwarzschild metric.
New method stops experiments early for harm in diverse groups.
problem Early stopping of experiments for harmful treatment effects in diverse populations.
method Causal machine learning approach (CLASH) for early stopping.
result CLASH effectively stops experiments early for harmful treatment effects in diverse groups.
Study on curvature decay in steady Ricci solitons, proving dichotomy.
problem Curvature decay in steady Ricci solitons.
method Established a dichotomy for curvature decay in specific types of solitons.
result Proved a dichotomy on curvature decay for certain steady Ricci solitons.
Cautious Weight Decay modifies weight decay for better optimization.
problem Improving optimization in deep learning models.
method Applies weight decay selectively based on parameter sign alignment.
result Consistently improves model performance across various tasks and scales.
This paper improves neural network predictions with early stopping using conformal calibration.
problem Lack of precise statistical guarantees for neural networks trained with early stopping.
method Conformalized early stopping that combines early stopping with conformal calibration.
result Models provide both accuracy and precise inferences without additional data splits.
Study on scalar curvature decay on non-compact manifolds linked at infinity.
problem Understanding scalar curvature decay on non-compact manifolds with topological linking at infinity.
method Analyzing polynomial decay, developing obstruction theory, using μ μ μ --bubble exhaustions, and index theory. result Topological linking at infinity forces polynomial decay of scalar curvature on manifolds of weakly bounded geometry.
Study on massless Vlasov equation on Reissner-Nordström spacetimes, showing decay rates and non-decay phenomena.
problem Analyzing decay and non-decay rates of solutions to the massless Vlasov equation on Reissner-Nordström spacetimes.
method Quantitative analysis of geodesic flow and comparison to wave equation instability results.
result Exponential decay rates in subextremal cases and polynomial rates in extremal cases, with non-decay of transversal derivatives in extremal cases.
Introduces gradient decay in Softmax for better generalization.
problem Improving generalization performance in neural networks.
method Gradient decay hyperparameter in Softmax for varying gradient rates based on probability.
result Gradient decay rate affects generalization performance and can be tuned for better optimization.
Study examines wave equation decay and Strichartz estimates on conic manifolds.
problem Analyzing wave equation behavior on conic spaces with critical electromagnetic potentials.
method Established decay and Strichartz estimates through localized spectral measure construction.
result Extended and improved previous results on wave equation behavior with critical potentials.
Gradient descent with early stopping achieves optimal sparse recovery.
problem Sparse regression with gradient descent and early stopping.
method Gradient descent on depth-N networks with early stopping.
result Implicit sparse regularization occurs with early stopping for general depth N.
Study shows uniform decay rate for singular mean curvature flows.
problem Understanding singularities in mean curvature flows.
method Rescaled flow analysis near compact singularities.
result Uniform decay order bound for the rescaled flow.
Weight decay stabilizes training dynamics by slowing progressive sharpening.
problem Understanding how weight decay affects training stability in deep learning models.
method Analyzing weight decay effects at the Edge of Stability, developing a mathematical framework.
result Weight decay dampens oscillations and stabilizes sharpness in CNNs, causing a phase transition in MLPs.
Jointly tuning ensemble models improves performance and uncertainty calibration.
problem Improving both predictive performance and uncertainty calibration in deep ensembles.
method Investigated the impact of jointly tuning weight decay, temperature scaling, and early stopping.
result Jointly tuning ensemble models generally matches or improves performance, with significant variation across tasks.
Polynomial decay of correlations shown for curved surfaces.
problem Analyzing geodesic flows on curved surfaces.
method Proving polynomial decay of correlations for geodesic flows on nonpositively curved surfaces.
result Polynomial decay of correlations for geodesic flows on nonpositively curved surfaces.
Early stopping methods reduce unnecessary reasoning steps in LLMs by monitoring uncertainty signals.
problem LLMs sometimes generate unnecessary reasoning steps, especially under uncertainty.
method Statistically principled early stopping methods that monitor uncertainty signals during generation.
result Uncertainty-aware early stopping improves efficiency and reliability in LLM reasoning, especially in math reasoning.
Paper analyzes pricing model for bonds with early redemption.
problem Analyzing pricing of bonds with early redemption features.
method Structural approach for mathematical modeling of bond prices.
result Existence and uniqueness of default and early redemption boundaries proved.
Last SGD iterate bounds for overparameterized linear regression.
problem Analyzing the last iterate risk bounds of SGD with decaying stepsize for overparameterized linear regression.
method Problem-dependent analysis of last iterate risk bounds of SGD with geometrically decaying stepsize.
result Proved nearly matching upper and lower bounds on the excess risk for last iterate SGD with geometrically decaying stepsize.
The study reveals optimal early stopping behaviors in deep learning models.
problem Understanding optimal early stopping in deep learning models.
method Theoretical analysis of linear models and experimental validation.
result Two distinct behaviors of optimal early stopping time depending on model dimension relative to dataset features.
This paper reviews early time series classification methods.
problem Minimizing class prediction delay in time-sensitive applications.
method Divided into four categories: prefix based, shapelet based, model based, and miscellaneous approaches.
result Demonstrates reasonable performance in various applications.
Optimal learning rates decay to zero in easy tasks and maintain a warmup phase in hard tasks.
problem Optimizing learning rates under functional scaling laws for model training.
method Deriving optimal learning-rate schedules based on exponents s s s and β β β . result Sharp phase transition between easy and hard tasks, with different decay behaviors.
We found that factors decay over time, with momentum fitting best.
problem Understanding how factors decay over time and their impact on performance.
method Derived a hyperbolic decay model for factors, tested against linear and exponential alternatives.
result Momentum exhibits hyperbolic decay, outperforming linear and exponential models.