Neural networks with Xavier initialization converge to global minimum in the scaling limit.
problem Optimizing neural networks with Xavier initialization in the large network limit.
method Stochastic analysis and convergence to a random ODE with a Gaussian distribution.
result The neural network converges to a global minimum in the limit, with zero loss.
Single layer Feedforward Neural Network(FNN) is used many a time as a last layer in models such as seq2seq or could be a simple RNN network. The importance of such layer is to transform the output to our required dimensions. When it comes to weights and biases initialization, there is no such specific technique that co…
Investigates scaling deep neural networks to avoid capacity issues.
problem Avoiding the shattering problem in deep neural networks.
method Formulates conjecture and studies various architectures to determine scaling relations.
result Reveals scaling relations for deep residual networks and recurrent networks.
New method for initializing RBM weights without datasets.
problem No dataset-free weight-initialization for RBMs.
method Statistical mechanical analysis to derive Gaussian distribution with optimized standard deviation.
result Optimal weight initialization improves learning efficiency in RBMs.
Deep neural networks (DNNs) form the backbone of almost every state-of-the-art technique in the fields such as computer vision, speech processing, and text analysis. The recent advances in computational technology have made the use of DNNs more practical. Despite the overwhelming performances by DNN and the advances in…
Gradient descent converges globally in deep linear residual networks with ZAS initialization.
problem Optimizing deep linear residual networks for convergence.
method Zero-asymmetric (ZAS) initialization for gradient descent.
result Gradient descent converges to an ε-optimal point in O(L^3 log(1/ε)) iterations.
Our study analyzes how neural network initialization affects privacy and utility in overparameterized models.
problem Privacy and utility trade-off in overparameterized neural networks.
method Analytical proof of KL divergence privacy bound, focusing on initialization, width, and depth.
result Privacy bound improvement with increasing depth under certain initializations, degradation under others.
Spectral normalization stabilizes GANs by controlling gradient explosion and vanishing.
problem Stability and sample quality issues in GAN training.
method Spectral normalization controls gradient explosion and vanishing, improving GAN training stability and sample quality.
result Bidirectional Scaled Spectral Normalization (BSSN) outperforms standard spectral normalization in sample quality and training stability.
S-GAI initializes MLPs using spectral geometry from data, improving performance.
problem Lack of guidance on initial weights encoding data geometry.
method S-GAI uses SVD to estimate spectral class geometry, initializing MLPs from training data.
result S-GAI-initialized MLPs start from a more informative hidden state and achieve comparable accuracy.
Neural networks trained with Q-learning converge to optimal control.
problem Optimal control of reinforcement learning with neural networks.
method Analysis of a limit differential equation derived from Q-learning.
result The limit differential equation has a unique stationary solution that is optimal.
The Residual Network (ResNet), proposed in He et al. (2015), utilized shortcut connections to significantly reduce the difficulty of training, which resulted in great performance boosts in terms of both training and generalization error. It was empirically observed in He et al. (2015) that stacking more layers of resid…
In this paper we will prove the Calabi-Yau conjectures for embedded surfaces. In fact, we will prove considerably more. The Calabi-Yau conjectures about surfaces date back to the 1960s. Much work has been done on them over the past four decades. In particular, examples of Jorge-Xavier from 1980 and Nadirashvili from 19…
New method improves NN performance across various settings.
problem Improving neural network performance across different datasets and architectures.
method Population Gradients (PG) method to calculate non-local gradient estimates.
result Significantly improves final performance across architectures, data-sets, and hyper-parameters.
Gradient descent dynamics in neural networks show quenching and activation phases.
problem Understanding training dynamics in neural networks.
method Numerical and phenomenological study of gradient descent algorithm for two-layer neural networks.
result Gradient descent dynamics exhibit quenching and activation phases in under-parametrized networks.
We consider topological conditions under which a locally invertible map admits a global inverse. Our main theorem states that a local diffeomorphism f:M→Rn is bijective if and only if Hn−1(M)=0 and the pre-image of every affine hyperplane is non-empty and acyclic. The proof is based on some geometr…
Study L2-harmonic forms on almost Kähler manifolds, extending vanishing theorems.
problem Analyzing L2-harmonic forms on complete almost Kähler manifolds. method Decomposing L2-harmonic forms into Lefschetz powers of primitive forms, extending vanishing theorems. result Spaces of harmonic (p,q)-forms on X vanish unless p+q=n. New algorithm learns random neural networks efficiently.
problem Learning random constant-depth neural networks efficiently.
method Presented a PTAS (Polynomial-Time Approximation Scheme) for learning random Xavier networks of fixed depth.
result For any fixed ε and depth i, there is a poly-time algorithm that learns random Xavier networks up to an additive error of ε.
Let Mn⊂Rn+1 be the graph of a C2-real valued function defined in a closed ball of Rn. In this work, we obtain upper bounds for infM∣H∣ and infM∣R∣, where H and R are, respectively, the mean curvature and the scalar curvature of Mn, generalizing estimates given by Heinz i…
Generalizing a theorem of Huang, Cheng and Wan classified the complete hypersurfaces of R4 with non-zero constant mean curvature and constant scalar curvature. In our work, we obtain results of this nature in higher dimensions. In particular, we prove that if a complete hypersurface of R5 has cons…
In this paper, we investigate the relationship between the discreteness of the spectrum of a non-compact, extrinsically bounded submanifold $\varphi \colon M^m \ra N^n$ and the Hausdorff dimension of its limit set limφ. In particular, we prove that if $\varphi \colon \!M^2 \ra D \subseteq \R^3$ is a minimal im…
Minimal surfaces with uniform curvature (or area) bounds have been well understood and the regularity theory is complete, yet essentially nothing was known without such bounds. We discuss here the theory of embedded (i.e., without self-intersections) minimal surfaces in Euclidean 3-space without a priori bounds. The st…
Paper optimizes neural network initialization using SMT solvers.
problem Improving neural network performance through better initialization.
method Reduces initialization to SMT problem solving.
result Proposed method achieves better performance than random initialization.
The paper explores neural network initialization strategies and proposes a new method.
problem The initialization phase of neural networks is underexplored compared to optimization.
method The authors discuss and develop an alternative initialization strategy for DNNs with ReLU activations.
result The new initialization strategy improves the quality of neural network training.
New method for better initial centers in clustering with improved accuracy and privacy.
problem Improving the quality of clustering centers in metric spaces.
method HST initialization based on metric embedding tree structure, combined with efficient search algorithm and DP extension.
result HST initialization produces better initial centers than k-median++ with comparable efficiency and improved privacy. Study shows different initialization schemes for LoRA finetuning impact performance.
problem The impact of initialization schemes on LoRA finetuning performance.
method Compared two initialization schemes: B=0, A=random vs. A=0, B=random.
result First initialization scheme yields better performance on average.
Orthogonal initialization speeds up convergence in deep linear networks.
problem The impact of initialization on convergence speed and model performance in deep neural networks.
method Analysis of orthogonal initialization in deep linear networks, proving its superiority over Gaussian initialization.
result Orthogonal initialization speeds up convergence relative to Gaussian initialization in deep networks.
New method learns good initialization for gradient descent from past solutions.
problem Challenges in solving non-convex optimization problems.
method Learning good initialization rules from previous solutions.
result Our approach performs better than random initialization in various non-convex problems.
Initial data for pp-wave spacetimes constructed in 4D.
problem Characterizing initial data for pp-wave spacetimes. method Constructs a vacuum initial data set with extra conditions related to CKID.
result Data development is a subset of a vacuum pp-wave. The paper establishes principles for initializing and designing GNNs with ReLU activations to avoid oversmoothing and correlation collapse.
problem Oversmoothing and correlation collapse in deep ReLU GNNs.
method The paper derives and validates three principles for initialization and architecture selection in finite width graph neural networks with ReLU activations.
result Correct initialization, residual aggregation operators, and residual connections significantly improve early training dynamics in deep ReLU GNNs.
In this paper, we firstly give a brief introduction of expectation maximization (EM) algorithm, and then discuss the initial value sensitivity of expectation maximization algorithm. Subsequently, we give a short proof of EM's convergence. Then, we implement experiments with the expectation maximization algorithm (We im…
The performance of the Self-Organizing Map (SOM) algorithm is dependent on the initial weights of the map. The different initialization methods can broadly be classified into random and data analysis based initialization approach. In this paper, the performance of random initialization (RI) approach is compared to that…
New findings show that minimal feature diversity at neural network initialization is harmful but can be mitigated with noise.
problem The importance of feature diversity at neural network initialization.
method A series of experiments comparing different initialization schemes, including adding noise.
result Minimal feature diversity is harmful but can be mitigated with noise, even standard GPU noise is sufficient.
New distributed clustering algorithms show resilience to initialization issues.
problem Resilience of distributed gradient-based clustering algorithms to center initialization.
method Distributed gradient-based clustering algorithms with novel center initialization.
result The algorithms are more resilient to initialization compared to baseline methods.
The paper proves deep ReLU networks can die and proposes a new initialization method to prevent it.
problem Dying ReLU neurons in deep neural networks.
method The paper rigorously proves the dying ReLU problem and proposes a new randomized asymmetric initialization method.
result The new initialization method effectively prevents the dying ReLU problem.
Barren plateaus are not an average-case phenomenon, but a highly non-unique problem.
problem Avoiding barren plateaus in neural network training
method First-moment framework for initialization strategies
result Many families of inequivalent initialization strategies can avoid concentration
New method for initializing networks robustly, improving performance across various settings.
problem Insufficient initialization strategies for weight-normalized networks and ResNets.
method Proposes a novel initialization strategy based on mean field approximation.
result Improves generalization performance and robustness to hyper-parameters.
Class I CR manifolds have initial G-structure a certain 4-dimensional subgroup of GL_3(C). Class II CR manifolds have initial G-structure a certain 10-dimensional subgroup of GL_4(C). Class III-1 CR manifolds have initial G-structure a certain 10-dimensional subgroup of GL_5(C). Class III-2 CR manifolds have initial G-…
Winning ticket initializations generalize across various datasets and optimizers.
problem Finding winning ticket initializations is computationally expensive and their generality is unclear.
method Generated winning tickets for one training configuration and evaluated their performance on another configuration.
result Winning ticket initializations generalize across multiple datasets and optimizers, achieving performance close to winning tickets generated on the same dataset.
FuseInit optimizes neural network initialization by fusing layers from deeper networks.
problem Initialization sensitivity of neural networks.
method Fusing neighboring layers from deeper networks trained with random initialization.
result Shallower networks initialized with FuseInit can perform as well as deeper networks.
Paper proves rigidity of initial data sets with boundary and capillary MOTS.
problem Rigidity of initial data sets with boundary and capillary MOTS.
method Estimates area of MOTS, proves rigidity for 3D, extends to high dimensions using Yamabe constant.
result Rigidity results for initial data sets with boundary and capillary MOTS.
Deep networks retain initial bias after training, affecting generalization.
problem Understanding how much initial bias in neural networks survives training.
method Introduced initialization memory to measure initial bias's survival.
result SGD can preserve initial bias, while Adam-family methods erase it.
AutoInit automatically finds good neural network initialization.
problem Finding optimal neural network initialization is crucial but time-consuming.
method Uses Jacobian tuning to automatically adjust network hyperparameters.
result The method finds good initialization for various network architectures.
This study examines the space of initial values strictly satisfying the dominant energy condition.
problem The dominant energy condition on initial value pairs in spacetime.
method Introduced an index difference for initial value pairs and compared it to Riemannian metrics.
result The space of initial values has non-trivial homotopy groups.
Study evaluates initialization strategies for infinite hidden Markov models.
problem Limited attention to initialization in infinite hidden Markov models.
method Systematically evaluated distance-based clustering, model-based, and uniform initializations.
result Distance-based clustering initializations consistently outperform other methods.
Neural nets trained with linear discriminant initialization converge faster and more accurately.
problem Training feed-forward neural networks efficiently and accurately.
method Initialize first layer weights with linear discriminants.
result Asymptotic higher accuracy and faster convergence.
This paper presents a phase diagram for two-layer neural networks under different initialization scales.
problem Understanding the behavior of neural networks under varying scales of initialization.
method Analysis of a phase diagram for two-layer neural networks.
result Condensation of weight vectors on isolated orientations during training.
Proves density and mass theorems for specific initial data sets.
problem Initial data sets with boundary in spacetime.
method Harmonic asymptotics and dominant energy condition.
result Spacetime positive mass theorem for initial data sets with apparent horizon boundary.
A new neural network initialization method is proposed for faster and more accurate training.
problem Efficient initialization for training multi-layer feedforward neural networks.
method Initialization based on Stein's identity, using eigenvectors of cross-moment matrix.
result The SteinGLM method is faster and more accurate than other initialization methods.