GD with large init shows incremental learning in matrix factorization.
problem Understanding GD's behavior with large initial values in matrix factorization.
method Signal-to-noise ratio concepts and inductive arguments.
result Uncovering an incremental learning phenomenon in GD with large initialization.
Two new scalable K-means initialization methods proposed for large-scale clustering.
problem Efficient initialization for large-scale clustering problems.
method Divide-and-conquer approach and random projection method for multiple lower-dimensional subspaces.
result The proposed methods outperform state-of-the-art in large-scale clustering tasks.
Develops path integral for spiked tensor model dynamics.
problem Dynamics of spiked tensor model with random initial conditions.
method Path integral approach applied to partial differential equations.
result Large-N saddle point equations dominated by melonic diagrams. Stochastic gradient descent with a large initial learning rate is widely used for training modern neural net architectures. Although a small initial learning rate allows for faster training and better test performance initially, the large learning rate achieves better generalization soon after the learning rate is anne…
Initializing the weights and the biases is a key part of the training process of a neural network. Unlike the subsequent optimization phase, however, the initialization phase has gained only limited attention in the literature. In this paper we discuss some consequences of commonly used initialization strategies for va…
Gradient descent with large steps leads to chaotic parameter space and unpredictable outcomes.
problem Understanding the behavior of gradient descent with large step sizes in matrix factorization.
method Analyzing the fractal structure of the parameter space and deriving critical step sizes for convergence.
result Gradient descent with large steps exhibits chaotic behavior and sensitivity to initialization, creating a fractal boundary between converging and diverging minimizers.
New method improves BO's AF maximizer initialization for high-dimensional problems.
problem Challenges in maximizing acquisition functions in high-dimensional Bayesian optimization.
method Proposes a heuristic optimizer-based initialization approach to improve AF maximizer performance.
result Our approach significantly enhances BO performance in most test cases.
We address the question of how stock prices respond to changes in demand. We quantify the relations between price change G over a time interval Δt and two different measures of demand fluctuations: (a) Φ, defined as the difference between the number of buyer-initiated and seller-initiated trades, and (b) Ω, def…
Large learning rates lead to optimal generalization if chosen carefully.
problem Understanding the optimal range of large learning rates for neural network training.
method Empirical study focusing on two questions: optimal initial LR range and differences between models trained with different LRs.
result Optimal initial learning rates slightly above the convergence threshold lead to optimal results after fine-tuning with a small LR or weight averaging.
This paper studies large-width asymptotics for ReLU neural networks with α-Stable initializations.
problem Characterizing the large-width behavior of ReLU neural networks with α-Stable initializations.
method Analysis of the large-width distributions and training dynamics of ReLU neural networks initialized with α-Stable distributions.
result For ReLU neural networks with α-Stable initializations, the large-width training dynamics achieve zero training error at a linear rate, characterized by a random kernel.
Improved LLM pre-training performance through better weight and variance control.
problem Improper weight and variance control in LLM pre-training affects downstream task performance.
method Introduced Layer Index Rescaling (LIR) and Target Variance Rescaling (TVR) techniques.
result Substantial improvements in downstream task performance (up to 4.6%) and reduced extreme activation values.
We carry out "exotic gluings" a la Carlotto-Schoen for asymptotically hyperbolic general relativistic initial data sets. In particular we obtain a direct construction of non-trivial initial data sets which are exactly hyperbolic in large regions extending to conformal infinity.
We give examples of asymptotically flat three-manifolds (M,g) which admit arbitrarily large constant mean curvature spheres that are far away from the center of the manifold. This resolves a question raised by G. Huisken and S.-T. Yau in 1996. On the other hand, we show that such surfaces cannot exist when (M,g) ha…
Constructs foliations of critical surfaces for Hawking energy in asymptotically flat initial data sets.
problem Positivity and rigidity of Hawking quasi-local energy in asymptotically flat spacetimes.
method Lyapunov-Schmidt reduction within a Willmore-foliation framework.
result Existence and uniqueness of foliations by Hawking surfaces, positivity and large-sphere limit of Hawking energy.
The K-means algorithm is a widely used clustering algorithm that offers simplicity and efficiency. However, the traditional K-means algorithm uses the random method to determine the initial cluster centers, which make clustering results prone to local optima and then result in worse clustering performance. Many initial…
Paper proposes efficient and accurate initialization and EM algorithm for PL mixture models.
problem Initialization issues and combinatorial complexity in PL likelihood maximization.
method Initialization algorithm and EM algorithm for true log-likelihood maximization.
result Proposed algorithm provides accurate initial estimates and efficiently maximizes true log-likelihood.
The common wisdom argues that, in general, large trades cause large price changes, while small trades cause small price changes. However, for extremely large price changes, the trade size and news play a minor role, while the liquidity (especially price gaps on the limit order book) is a more influencing factor. Hence,…
Bounds neural network output distribution to Gaussian for random initialization.
problem Quantifying the distribution of randomly initialized deep neural networks.
method Quantitative Gaussian approximation using quadratic Wasserstein distance.
result Explicit inequalities show how network sizes affect Gaussian behavior.
Single neural network predicts ImageNet model parameters for faster training.
problem Training diverse ImageNet models requires significant resources and time.
method Trained a neural network to predict ImageNet model parameters and used them for initialization.
result Models initialized with predicted parameters converge faster and achieve competitive performance.
Study shows how 3+1D cosmologies can evolve to de Sitter space under certain conditions.
problem Understanding the evolution of 3+1D cosmologies with specific symmetry constraints.
method Mean Curvature Flow methods applied to cosmologies with positive cosmological constant and specific symmetry groups.
result Asymptotically, 3+1D cosmologies evolve to de Sitter space under certain conditions.
Orthogonal initialization does not speed up training in ultra-wide neural networks.
problem Exploring the effect of orthogonal initialization on training speed in deep neural networks.
method Study of neural tangent kernel dynamics in FCNs and CNNs with orthogonal initialization.
result The NTK of orthogonally-initialized networks remains constant during training, suggesting no speedup in the NTK regime.
We analyze the global convergence of gradient descent for deep linear residual networks by proposing a new initialization: zero-asymmetric (ZAS) initialization. It is motivated by avoiding stable manifolds of saddle points. We prove that under the ZAS initialization, for an arbitrary target matrix, gradient descent con…
Randomly initialized transformers show extreme token preferences.
problem Structural biases in randomly initialized transformers.
method Dissection of transformer architecture at initialization.
result Initialization-induced biases persist throughout training.
Paper introduces scalable clustering for large datasets with outliers.
problem Lack of scalable algorithms for large datasets with outliers.
method Provable robust clustering algorithm based on loss minimization for Gaussian mixture models.
result Algorithm provides high accuracy with theoretical guarantees and outperforms existing methods.
The paper establishes principles for initializing and designing GNNs with ReLU activations to avoid oversmoothing and correlation collapse.
problem Oversmoothing and correlation collapse in deep ReLU GNNs.
method The paper derives and validates three principles for initialization and architecture selection in finite width graph neural networks with ReLU activations.
result Correct initialization, residual aggregation operators, and residual connections significantly improve early training dynamics in deep ReLU GNNs.
Maximal initial learning rate for deep ReLU networks identified.
problem Finding the optimal initial learning rate for deep neural networks.
method Simple approach to estimate maximal initial learning rate η∗, analyzing its behavior in constant-width fully-connected ReLU networks. result Maximal initial learning rate η∗ is well predicted as a power of depth × width, with specific conditions for network width and input layer training. We prove the existence of a large class of initial data for the vacuum Einstein equations which possess a finite number of asymptotically Euclidean and asymptotically conformally cylindrical or periodic ends. Aside from being asymptotically constant, only mild conditions on the mean curvature of these initial data sets…
The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.
problem Understanding training dynamics in deep linear networks from random initialization.
method Theoretical analysis of gradient descent dynamics in deep linear networks with random initialization and large data.
result Captures the 'wider is better' effect and hyperparameter transfer effects, contrasting with neural-tangent parameterization.
New method improves convergence of spatial filters in neural networks.
problem Poor convergence behavior of spatial filters in neural networks.
method Correlated initialization for spatial filters.
result Uncorrelated initialization leads to poor convergence and slow training of some parameters.
Public debt is one of the important economic variables that quantitatively describes a nation's economy. Because bankruptcy is a risk faced even by institutions as large as governments (e.g. Iceland), national debt should be strictly controlled with respect to national wealth. Also, the problem of eliminating extreme p…
New proof links initial class bias to DNN trainability, challenging traditional understanding.
problem Understanding the initial class bias in DNNs and its impact on trainability.
method Theoretical proof linking initial class bias to mean field theories of DNNs.
result Efficient learning is connected to a network's prejudice towards a specific class, contradicting traditional understanding.
Logarithmic pruning simplifies lottery ticket hypothesis.
problem Finding efficient subnetworks in large neural networks.
method Logarithmic pruning approach to identify subnetworks.
result Randomly initialized subnetworks achieve comparable performance.
New method stabilizes deep neural networks by setting Lyapunov exponent to zero.
problem Stability issues in deep neural networks with low width.
method Lyapunov initialization method to set Lyapunov exponent to zero.
result Lyapunov exponent governs stability of deep networks; standard methods fail for low width.
We construct large families of initial data sets for the vacuum Einstein equations with positive cosmological constant which contain exactly Delaunay ends; these are non-trivial initial data sets which coincide with those for the Kottler-Schwarzschild-de Sitter metrics in regions of infinite extent. From the purely Rie…
The success of lottery ticket initializations (Frankle and Carbin, 2019) suggests that small, sparsified networks can be trained so long as the network is initialized appropriately. Unfortunately, finding these "winning ticket" initializations is computationally expensive. One potential solution is to reuse the same wi…
In the first half of this article, we survey the new quasi-local and total angular momentum and center of mass defined in [9] and summarize the important properties of these definitions. To compute these conserved quantities involves solving a nonlinear PDE system (the optimal isometric embedding equation), which is ra…
Global properties of maximal future Cauchy developments of stationary, m-dimensional asymptotically flat initial data with an outer trapped boundary are analyzed. We prove that, whenever the matter model is well posed and satisfies the null energy condition, the future Cauchy development of the data is a black hole spa…
Two-layer CNNs can overfit well if initialized correctly.
problem Understanding the conditions for benign overfitting in over-parameterized CNNs.
method Extending analysis to fully trainable two-layer CNNs, examining initialization scaling effects.
result Initialization scaling of the output layer is crucial; large scales lead to fixed output behavior, small scales to complex interactions.
Deep linear networks minimize sharpness, avoiding large eigenvalues.
problem Understanding optimization dynamics in deep linear networks for regression.
method Analyzing sharpness (largest eigenvalue of Hessian) of minimizers and gradient flow solutions.
result Gradient flow implicitly regularizes towards flat minima, with sharpness bounded by a constant.
Our study analyzes how neural network initialization affects privacy and utility in overparameterized models.
problem Privacy and utility trade-off in overparameterized neural networks.
method Analytical proof of KL divergence privacy bound, focusing on initialization, width, and depth.
result Privacy bound improvement with increasing depth under certain initializations, degradation under others.
Constructs initial data for Einstein vacuum equations involving multiple localized gravitational sources.
problem Modeling the interaction of distant gravitational systems in general relativity.
method Time-symmetric initial data construction using gluing schemes and localized sources.
result Produces initial data sets with finite ADM mass and multiple Einstein-Rosen bridges.
Solves Ricci flow on Riemann surfaces with measure initial data.
problem Existence and smoothness of Ricci flow on Riemann surfaces.
method Formulation and solution of existence problem using Ricci flow.
result New examples of nongradient expanding Ricci solitons.
Large stepsize GD for logistic regression converges faster than expected.
problem Optimizing logistic regression with large step sizes.
method Gradient descent with large stepsize applied to logistic regression.
result GD converges to a lower loss in fewer steps than expected.
MCE reduces embedding instability in nonlinear dimensionality reduction.
problem Embedding instability caused by random initialization.
method Median of multiple embeddings (MCE) based on large deviation theory.
result MCE achieves consistency at an exponential rate and effectively mitigates instability.
We propose a novel time discretization for the log-normal SABR model which is a popular stochastic volatility model that is widely used in financial practice. Our time discretization is a variant of the Euler-Maruyama scheme. We study its asymptotic properties in the limit of a large number of time steps under a certai…
Deep networks retain initial bias after training, affecting generalization.
problem Understanding how much initial bias in neural networks survives training.
method Introduced initialization memory to measure initial bias's survival.
result SGD can preserve initial bias, while Adam-family methods erase it.
Mimetic initialization improves Transformer training on small datasets.
problem Difficulty in training Transformers on small datasets.
method Initialize self-attention layers to look like pre-trained models.
result Vanilla Transformers trained with mimetic initialization achieve higher accuracy.
AMP method reconstructs rank-one matrices from noisy data efficiently.
problem Reconstructing rank-one matrices with prior structural information from noisy observations.
method Approximate Message Passing (AMP) with random initialization.
result AMP from random initialization converges rapidly and globally.