The performance of the Self-Organizing Map (SOM) algorithm is dependent on the initial weights of the map. The different initialization methods can broadly be classified into random and data analysis based initialization approach. In this paper, the performance of random initialization (RI) approach is compared to that…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,738 papers · 148 categories
Trend · papers per month
48 results for “random initializations”
GNNs with random node initialization are shown to be universally expressive.
problem Limitations of standard GNNs in distinguishing graphs.
method Random node initialization (RNI) to enhance GNNs' expressive power.
result GNNs with RNI are proven to be universally expressive.
Beyond Signal Propagation: Is Feature Diversity Necessary in Deep Neural Network Initialization?cs.LG
New method shows random, diverse initializations are not essential for deep neural networks.
problem The necessity of random, diverse initializations in deep neural networks.
method Constructed a deep convolutional network with identical features by initializing weights to 0, enabling signal propagation and stable gradients.
result Random, diverse initializations are not necessary for training neural networks.
This paper examines the impact of random initialization in neural networks using NTK theory.
problem Understanding the impact of random initialization in neural networks using NTK theory.
method Analyzes the convergence of training dynamics and generalization error of wide neural networks with random initialization.
result The generalization error of wide neural networks trained by gradient descent is \( \Omega(n^{-\frac{3}{d+3}}) \), highlighting the benefits of mirror initialization and suggesting limitations of NTK theory.
Randomly initialized transformers show extreme token preferences.
problem Structural biases in randomly initialized transformers.
method Dissection of transformer architecture at initialization.
result Initialization-induced biases persist throughout training.
Bounds neural network output distribution to Gaussian for random initialization.
problem Quantifying the distribution of randomly initialized deep neural networks.
method Quantitative Gaussian approximation using quadratic Wasserstein distance.
result Explicit inequalities show how network sizes affect Gaussian behavior.
Gradient Descent with small random initialization solves rank-1 matrix completion efficiently.
problem Matrix completion for rank-1 symmetric matrices.
method Gradient Descent with small random initialization.
result Gradient Descent converges to the ground truth for rank-1 symmetric matrix completion.
Develops path integral for spiked tensor model dynamics.
problem Dynamics of spiked tensor model with random initial conditions.
method Path integral approach applied to partial differential equations.
result Large- saddle point equations dominated by melonic diagrams.
This paper improves privacy bounds for DP algorithms using -DP.
problem Difficulty in analyzing randomness in DP algorithms due to mixture distributions.
method Derives a closed-form expression for trade-off functions and analyzes -DP.
result Enhances privacy of DP-GD with random initialization and shuffling models.
Two new scalable K-means initialization methods proposed for large-scale clustering.
problem Efficient initialization for large-scale clustering problems.
method Divide-and-conquer approach and random projection method for multiple lower-dimensional subspaces.
result The proposed methods outperform state-of-the-art in large-scale clustering tasks.
Recent pruning methods at initialization fall short of random pruning's accuracy.
problem Improving neural network accuracy through pruning at initialization.
method Various pruning methods (SNIP, GraSP, SynFlow, magnitude pruning) are evaluated; per-layer pruning decisions are proposed.
result Randomly shuffling or sampling initial weights preserves or improves accuracy, suggesting challenges with pruning heuristics.
FCNv2 robustness tested under noise and random initial conditions.
problem Assessing AI weather forecasting model robustness to input noise.
method Two experiments with varying noise levels and random initial conditions.
result FCNv2 preserves hurricane features under low to moderate noise, but underestimates intensity and persistence.
New method learns good initialization for gradient descent from past solutions.
problem Challenges in solving non-convex optimization problems.
method Learning good initialization rules from previous solutions.
result Our approach performs better than random initialization in various non-convex problems.
In this paper we study a model of random knots obtained by fixing a space curve in -dimensional Euclidean space with , and orthogonally projecting the space curve on to random dimensional subspaces. By varying the space curve we obtain different models of random parametrized knots, and we will study how the…
The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.
problem Understanding training dynamics in deep linear networks from random initialization.
method Theoretical analysis of gradient descent dynamics in deep linear networks with random initialization and large data.
result Captures the 'wider is better' effect and hyperparameter transfer effects, contrasting with neural-tangent parameterization.
Study on adversarial robustness in neural networks across initialization and training phases.
problem Understanding adversarial robustness in neural networks during different learning stages.
method Analyzes adversarial robustness in various scenarios of over-parameterized networks with quadratic targets and infinite samples.
result Robustness can worsen when test error improves, and vice versa, revealing new tradeoffs.
Randomly initialized neural networks can linearly separate arbitrary sets.
problem Mapping two arbitrary sets to linearly separable sets.
method Randomly initialized one-layer neural networks with sufficient width.
result With high probability, these networks can transform two sets into linearly separable sets.
AMP method reconstructs rank-one matrices from noisy data efficiently.
problem Reconstructing rank-one matrices with prior structural information from noisy observations.
method Approximate Message Passing (AMP) with random initialization.
result AMP from random initialization converges rapidly and globally.
Algorithm recovers factors of rank-1 matrices from noisy measurements.
problem Estimating factors of a rank-1 matrix from nonlinearly transformed and noisy measurements.
method Alternating minimization with random initialization and analysis of empirical error recursion.
result Algorithm converges geometrically fast from random initialization, with sharp guarantees.
Study shows different initialization schemes for LoRA finetuning impact performance.
problem The impact of initialization schemes on LoRA finetuning performance.
method Compared two initialization schemes: B=0, A=random vs. A=0, B=random.
result First initialization scheme yields better performance on average.
Bayesian method improves neural net convergence for character recognition.
problem Improving convergence rate of neural network training algorithms.
method Customization of Kalman filter into Bayesian statistics for initialization of weights.
result Improved convergence rate for backpropagation training algorithm.
FuseInit optimizes neural network initialization by fusing layers from deeper networks.
problem Initialization sensitivity of neural networks.
method Fusing neighboring layers from deeper networks trained with random initialization.
result Shallower networks initialized with FuseInit can perform as well as deeper networks.
In this paper, we study the problem of expected utility maximization of an agent who, in addition to an initial capital, receives random endowments at maturity. Contrary to previous studies, we treat as the variables of the optimization problem not only the initial capital but also the number of units of the random end…
A new method to prune neural networks with iterative randomization improves efficiency.
problem Efficiency in pruning randomly initialized neural networks.
method Iteratively randomizing weight values to reduce parameter requirements.
result The method achieves remarkable performance with fewer parameters.
In this paper, we present a novel approach for initializing deep neural networks, i.e., by turning PCA into neural layers. Usually, the initialization of the weights of a deep neural network is done in one of the three following ways: 1) with random values, 2) layer-wise, usually as Deep Belief Network or as auto-encod…
SGD fails to converge for deep ReLU networks with limited random initializations.
problem SGD convergence in deep neural networks with limited random initializations.
method Analysis of four discretization parameters: network architecture, training data, gradient steps, and random initializations.
result SGD fails to converge for ReLU networks with depth much larger than width.
Gradient descent with small random init mimics spectral methods for low-rank matrix recovery.
problem Reconstructing a low-rank matrix from few measurements.
method Gradient descent with small random initialization followed by a few iterations.
result Gradient descent from small random init converges to a well-generalizing solution.
Randomly initialized networks can perform as well as pruned networks.
problem Understanding and improving network pruning methods.
method Sanity checks on recent pruning methods, proposing random tickets.
result Randomly initialized networks can perform as well as pruned networks.
Deep learning relies on good initialization schemes and hyperparameter choices prior to training a neural network. Random weight initializations induce random network ensembles, which give rise to the trainability, training speed, and sometimes also generalization ability of an instance. In addition, such ensembles pro…
White matter hyperintensity (WMH) is commonly found in elder individuals and appears to be associated with brain diseases. U-net is a convolutional network that has been widely used for biomedical image segmentation. Recently, U-net has been successfully applied to WMH segmentation. Random initialization is usally used…
New method stabilizes deep neural networks by setting Lyapunov exponent to zero.
problem Stability issues in deep neural networks with low width.
method Lyapunov initialization method to set Lyapunov exponent to zero.
result Lyapunov exponent governs stability of deep networks; standard methods fail for low width.
We investigate i.i.d. random complex dynamical systems generated by probability measures on finite unions of the loci of holomorphic families of rational maps on the Riemann sphere. We show that under certain conditions on the families, for a generic system, (especially, for a generic random polynomial dynamical system…
MCE reduces embedding instability in nonlinear dimensionality reduction.
problem Embedding instability caused by random initialization.
method Median of multiple embeddings (MCE) based on large deviation theory.
result MCE achieves consistency at an exponential rate and effectively mitigates instability.
Paper optimizes neural network initialization using SMT solvers.
problem Improving neural network performance through better initialization.
method Reduces initialization to SMT problem solving.
result Proposed method achieves better performance than random initialization.
One of the fundamental problems in supervised classification and in machine learning in general, is the modelling of non-parametric invariances that exist in data. Most prior art has focused on enforcing priors in the form of invariances to parametric nuisance transformations that are expected to be present in data. Le…
We study the statistical properties of an estimator derived by applying a gradient ascent method with multiple initializations to a multi-modal likelihood function. We derive the population quantity that is the target of this estimator and study the properties of confidence intervals (CIs) constructed from asymptotic n…
Randomly initialized ReLU networks of depth two can approximate smooth functions well.
problem Approximation power of two-layer networks of random ReLUs.
method Harmonic analysis and ridgelet representation theory for upper bounds, dimensionality arguments for lower bounds.
result Near-matching upper and lower bounds for -approximation and Sobolev norms.
On the effects of biased quantum random numbers on the initialization of artificial neural networksquant-ph
Study finds no significant difference in neural network weights with quantum random numbers.
problem Effects of biased quantum random numbers on neural network initialization.
method Empirical study using quantum hardware and classical pseudo-random numbers.
result No statistically significant difference found between quantum random numbers and other types.
Randomly initialized networks contain subnetworks that perform similarly to target networks.
problem Proving the lottery ticket hypothesis for neural networks.
method Pruning over-parameterized neural networks to find subnetworks.
result Randomly initialized networks contain subnetworks with similar performance to target networks without additional training.
New method improves adversarial training efficiency and robustness.
problem High computational costs and lack of stability in adversarial training.
method Backward smoothing for randomized smoothing of random initialization.
result Our method achieves similar model robustness as state-of-the-art methods but with significantly less training time.
Study on the smoothness of solutions to a specific type of stochastic differential equation.
problem Regularity of solutions to mean-field -SDEs.
method Analysis of first and second order Fréchet differentiability in the random initial condition.
result Established the Fréchet differentiability of the solution and specified the corresponding equations.
Mixed datasets consist of both numeric and categorical attributes. Various k-means-based clustering algorithms have been developed for these datasets. Generally, these algorithms use random partition as a starting point, which tends to produce different clustering results for different runs. In this paper, we propose, …
New method improves model robustness against adversarial attacks.
problem Adversarial attacks can deceive models trained with transfer learning.
method Noisy feature distillation method for random initialization.
result Models trained with random initialization are more robust to attacks.
New hyperparameter ensembles boost neural network performance and uncertainty.
problem Improving neural network robustness and uncertainty quantification.
method Designing ensembles over both weights and hyperparameters, stratified across random initializations.
result Hyper-deep and hyper-batch ensembles outperform deep and batch ensembles on various architectures.
Gradient descent amplifies random features in neural networks to useful ones.
problem Generalization in neural networks trained on corrupted data.
method Characterization of feature-learning process in two-layer ReLU networks trained by gradient descent.
result Gradient descent amplifies random features to useful ones, achieving near optimal generalization error.
New algorithms use outsourced data to improve model training efficiency.
problem Limited computational resources restrict model training efficiency.
method Simulation-based algorithms using outsourced data to find good initial points.
result The algorithms can find good initial points with high probability under suitable conditions.
Despite the widespread practical success of deep learning methods, our theoretical understanding of the dynamics of learning in deep neural networks remains quite sparse. We attempt to bridge the gap between the theory and practice of deep learning by systematically analyzing learning dynamics for the restricted case o…
The paper confirms a conjecture about optimal expected utility in markets with insider information.
problem Optimal expected utility in markets with insider information.
method An extension of the Black-Scholes-Merton model with a sequence of discrete-time economies.
result Optimal expected utility converges to the classic model when conditions are met.