Paper optimizes neural network initialization using SMT solvers.
problem Improving neural network performance through better initialization.
method Reduces initialization to SMT problem solving.
result Proposed method achieves better performance than random initialization.
Maxout networks study gradients and propose initialization strategies.
problem Complexity in input-output Jacobian distribution complicates stable parameter initialization.
method Obtained bounds on moments of gradients and formulated initialization strategies.
result Parameter initialization strategies improve training of deep maxout networks.
Chameleon learns model initializations across tasks with different schemas.
problem Meta-learning parameter initialization across tasks with varying predictor schemas.
method Chameleon aligns different predictor schemas to a common representation.
result Chameleon successfully learns parameter initializations across tasks with different schemas.
Constructs initial data for multiple black holes with specified ADM parameters.
problem Forming multiple black holes with specific ADM parameters.
method Smooth, asymptotically flat vacuum initial data with prescribed ADM energy, momentum, and angular momentum.
result Maximal development of data results in spacetimes containing multiple black holes.
Residual networks (ResNet) and weight normalization play an important role in various deep learning applications. However, parameter initialization strategies have not been studied previously for weight normalized networks and, in practice, initialization methods designed for un-normalized networks are used as a proxy.…
Single neural network predicts ImageNet model parameters for faster training.
problem Training diverse ImageNet models requires significant resources and time.
method Trained a neural network to predict ImageNet model parameters and used them for initialization.
result Models initialized with predicted parameters converge faster and achieve competitive performance.
The paper introduces a multilevel initialization method for deep neural networks.
problem Training very deep neural networks with layer-parallel methods.
method Continuous interpretation of training as optimal control, using time-dependent ODEs for neural network discretization, and a refinement strategy across the time domain.
result The method creates deep networks with good initializations from coarser networks, reducing training time and providing regularization.
New method for initializing low-rank neural networks improves performance.
problem Training low-rank neural networks efficiently and accurately.
method Inspired by function approximation, proposes a novel low-rank initialization framework.
result Demonstrates significant gap between spectral and low-rank initialization approaches.
New method for initializing RBM weights without datasets.
problem No dataset-free weight-initialization for RBMs.
method Statistical mechanical analysis to derive Gaussian distribution with optimized standard deviation.
result Optimal weight initialization improves learning efficiency in RBMs.
FuseInit optimizes neural network initialization by fusing layers from deeper networks.
problem Initialization sensitivity of neural networks.
method Fusing neighboring layers from deeper networks trained with random initialization.
result Shallower networks initialized with FuseInit can perform as well as deeper networks.
Barren plateaus are not an average-case phenomenon, but a highly non-unique problem.
problem Avoiding barren plateaus in neural network training
method First-moment framework for initialization strategies
result Many families of inequivalent initialization strategies can avoid concentration
We investigate unification of two systems of identical elements having different dimensions which may be of interest for both physics and economics. Characteristic parameters as well as explicit formulae for the temperature (in economics - capital turnover) and dimension of the united system are obtained as functions o…
In this effort we propose a novel approach for reconstructing multivariate functions from training data, by identifying both a suitable network architecture and an initialization using polynomial-based approximations. Training deep neural networks using gradient descent can be interpreted as moving the set of network p…
New method stabilizes deep neural networks by setting Lyapunov exponent to zero.
problem Stability issues in deep neural networks with low width.
method Lyapunov initialization method to set Lyapunov exponent to zero.
result Lyapunov exponent governs stability of deep networks; standard methods fail for low width.
A new method to prune neural networks with iterative randomization improves efficiency.
problem Efficiency in pruning randomly initialized neural networks.
method Iteratively randomizing weight values to reduce parameter requirements.
result The method achieves remarkable performance with fewer parameters.
SGD fails to converge for deep ReLU networks with limited random initializations.
problem SGD convergence in deep neural networks with limited random initializations.
method Analysis of four discretization parameters: network architecture, training data, gradient steps, and random initializations.
result SGD fails to converge for ReLU networks with depth much larger than width.
Pruning at initialization fails to find sparse subnetworks, revealing information-theoretic barriers.
problem Difficulty in finding sparse subnetworks without training the full model.
method Analysis of effective parameter count and mutual information between sparsity mask and data.
result Pruning at initialization cannot find sparse subnetworks due to high mutual information.
In this paper, we firstly give a brief introduction of expectation maximization (EM) algorithm, and then discuss the initial value sensitivity of expectation maximization algorithm. Subsequently, we give a short proof of EM's convergence. Then, we implement experiments with the expectation maximization algorithm (We im…
Proposes a new pretraining strategy for RNNs to improve classification performance.
problem Poor generalization of RNNs due to initial parameter assignment.
method Data-aware layer-wise pretraining strategy to initialize RNN parameters.
result Data-aware strategies positively support the initialization of RNN-based classification models.
Memory-Augmented Meta-Optimization improves cold-start recommendation.
problem Cold-start problem in recommender systems for new users or items.
method Memory-Augmented Meta-Optimization approach with personalized and task-specific memories.
result Significant improvement in cold-start recommendation performance on multiple datasets.
New model reveals significant impact of data and parameter variations on machine learning benchmarks.
problem Variance in machine learning benchmarks due to data sampling, initialization, and hyperparameters.
method Modeling the benchmarking process, analyzing comparison methods, and studying error rates.
result Adding more sources of variation reduces compute cost while improving performance.
Gradient descent with large steps leads to chaotic parameter space and unpredictable outcomes.
problem Understanding the behavior of gradient descent with large step sizes in matrix factorization.
method Analyzing the fractal structure of the parameter space and deriving critical step sizes for convergence.
result Gradient descent with large steps exhibits chaotic behavior and sensitivity to initialization, creating a fractal boundary between converging and diverging minimizers.
HOPE improves SSMs for long-memory tasks with robust initialization and training.
problem Improving state-space models for long-memory tasks with robust initialization and training.
method Developed a new parameterization scheme called HOPE using Hankel operators and Markov parameters.
result HOPE improves SSMs' performance on Long-Range Arena tasks and demonstrates non-decaying memory.
Optimizes deep neural network initialization variance for better performance.
problem Improving deep neural network performance through optimal initialization variance.
method Using SGD dynamics and Fokker-Planck equations, we study the relationship between initialization and expected loss function.
result An optimal condition for initialization variance that leads to lower training loss and higher test accuracy.
Bounds neural network output distribution to Gaussian for random initialization.
problem Quantifying the distribution of randomly initialized deep neural networks.
method Quantitative Gaussian approximation using quadratic Wasserstein distance.
result Explicit inequalities show how network sizes affect Gaussian behavior.
This paper studies transformer learning dynamics and initialization.
problem Understanding how transformers learn Markov chains and the role of initialization.
method First-order Markov chains and single-layer transformers, proving learning dynamics and conditions for convergence.
result Transformer parameters can converge to global or local minima based on initialization and Markovian data properties.
The dying ReLU refers to the problem when ReLU neurons become inactive and only output 0 for any input. There are many empirical and heuristic explanations of why ReLU neurons die. However, little is known about its theoretical analysis. In this paper, we rigorously prove that a deep ReLU network will eventually die in…
Optimizes wide low-rank neural networks for reduced parameters and cost.
problem Reducing the number of learnable parameters in wide neural networks.
method Analyzed edge-of-chaos dynamics and derived formulae for optimal weight and bias variances.
result Optimal weight and bias variances for low-rank networks follow from multiplicative scaling.
EVA adapts LoRA for faster, more efficient fine-tuning.
problem Fast and efficient fine-tuning of large models for specific tasks.
method EVA uses directions capturing most activation variance for initialization, maximizing gradient signal and reducing parameters.
result EVA achieves faster convergence and higher average scores across tasks, reducing parameters.
Neural IVP solves IVPs with neural networks, overcoming scaling and conditioning issues.
problem Solving initial value PDEs with neural networks is challenging due to numerical errors and limited scalability.
method Developed an ODE-based approach to solve IVPs with neural networks, preventing ill-conditioning and scaling issues.
result Neural IVP solves challenging PDEs with neural networks efficiently and accurately.
Genus one singularity appears in mean curvature flow for certain initial conditions.
problem Understanding singularities in mean curvature flow.
method Analyzing one-parameter families of initial conditions in R3. result A robust genus one singularity appears in mean curvature flow.
In this article we provide a formulation of empirical bayes described by Atchade (2011) to tune the hyperparameters of priors used in bayesian set up of collaborative filter. We implement the same in MovieLens small dataset. We see that it can be used to get a good initial choice for the parameters. It can also be used…
The paper develops a theory linking pretraining and fine-tuning in neural networks.
problem Understanding how initialization choices impact feature learning and generalization in neural networks.
method Analytical theory of diagonal linear networks, deriving generalization error as a function of initialization parameters and task statistics.
result Different initialization choices place networks into four fine-tuning regimes with varying abilities to support feature learning and generalization.
Orthogonal initialization speeds up convergence in deep linear networks.
problem The impact of initialization on convergence speed and model performance in deep neural networks.
method Analysis of orthogonal initialization in deep linear networks, proving its superiority over Gaussian initialization.
result Orthogonal initialization speeds up convergence relative to Gaussian initialization in deep networks.
Wider neural networks perform better than deeper ones with the same number of parameters.
problem Understanding the role of network width versus the number of parameters in neural network performance.
method Comparing models with different ways of increasing width while keeping the number of parameters constant, analyzing their performance and using Gaussian Process kernels for analysis.
result Network width is the determining factor for good performance, while the number of weights is secondary as long as trainability is ensured.
We develop a gluing construction which adds scaled and truncated asymptotically Euclidean solutions of the Einstein constraint equations to compact solutions with potentially non-trivial cosmological constants. The result is a one-parameter family of initial data which has ordinary and scaled "point-particle" limits an…
We give convergence guarantees for estimating the coefficients of a symmetric mixture of two linear regressions by expectation maximization (EM). In particular, we show that the empirical EM iterates converge to the target parameter vector at the parametric rate, provided the algorithm is initialized in an unbounded co…
New method improves convergence of spatial filters in neural networks.
problem Poor convergence behavior of spatial filters in neural networks.
method Correlated initialization for spatial filters.
result Uncorrelated initialization leads to poor convergence and slow training of some parameters.
Method for initializing Gaussian mixtures for variational inference with multi-modal distributions.
problem Challenges in variational inference with Gaussian mixtures due to multimodality and nonconvex loss functions.
method Optimization to find local maxima, local Gaussian approximations, and constrained least squares regression.
result Robust initialization improves variational inference performance and scalability.
Initialization of parameters in deep neural networks has been shown to have a big impact on the performance of the networks (Mishkin & Matas, 2015). The initialization scheme devised by He et al, allowed convolution activations to carry a constrained mean which allowed deep networks to be trained effectively (He et al.…
New proof links initial class bias to DNN trainability, challenging traditional understanding.
problem Understanding the initial class bias in DNNs and its impact on trainability.
method Theoretical proof linking initial class bias to mean field theories of DNNs.
result Efficient learning is connected to a network's prejudice towards a specific class, contradicting traditional understanding.
PINNs solve neuronal parameter and state estimation problems with limited data.
problem Estimating parameters and hidden state variables from noisy partial data in multiscale neuronal models.
method Physics-informed neural networks (PINNs) for joint state and parameter estimation.
result PINNs deliver robust and accurate parameter inference and state reconstruction, even with limited data.
Why does training deep neural networks using stochastic gradient descent (SGD) result in a generalization error that does not worsen with the number of parameters in the network? To answer this question, we advocate a notion of effective model capacity that is dependent on {\em a given random initialization of the netw…
In this paper, we present a novel approach for initializing deep neural networks, i.e., by turning PCA into neural layers. Usually, the initialization of the weights of a deep neural network is done in one of the three following ways: 1) with random values, 2) layer-wise, usually as Deep Belief Network or as auto-encod…
Adaptive RBF-KAN improves KANs by dynamically adjusting kernel parameters.
problem Efficiently approximating multivariate functions using learnable univariate edge functions.
method Integrates LOOCV-based kernel scale estimation with adaptive kernel learning.
result Adaptive RBF-KAN outperforms fixed kernel KANs on various benchmark functions.
Simplified Butterfly-Net2 improves CNN efficiency in solving PDEs and signal processing tasks.
problem Improving CNN efficiency in solving PDEs and signal processing tasks.
method Introducing BNet2, a simplified Butterfly-Net, and Fourier transform initialization.
result BNet2 achieves similar accuracy as CNN but with fewer parameters and improves accuracy over randomly initialized CNN.
We explore how neural networks train to zero loss, focusing on initial scale.
problem Understanding neural network training dynamics and zero loss.
method Macroscopic limits analysis of gradient descent dynamics.
result Gradient descent can drive deep neural networks to zero loss regardless of initialization.
Deep learning relies on good initialization schemes and hyperparameter choices prior to training a neural network. Random weight initializations induce random network ensembles, which give rise to the trainability, training speed, and sometimes also generalization ability of an instance. In addition, such ensembles pro…