A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
New method for initializing RBM weights without datasets.
This paper examines weight initialization for 1-Lipschitz networks to improve robustness against adversarial attacks.
Bayesian method improves neural net convergence for character recognition.
Residual Network (ResNet) is the state-of-the-art architecture that realizes successful training of really deep neural network. It is also known that good weight initialization of neural network avoids problem of vanishing/exploding gradients. In this paper, simplified models of ResNets are analyzed. We argue that good…
Improved LLM pre-training performance through better weight and variance control.
Proposes a new method to initialize neural networks by estimating global curvature of weights.
Paper extends positive energy theorem to anti-de Sitter spacetimes.
Residual networks (ResNet) and weight normalization play an important role in various deep learning applications. However, parameter initialization strategies have not been studied previously for weight normalized networks and, in practice, initialization methods designed for un-normalized networks are used as a proxy.…
New method improves convergence of spatial filters in neural networks.
A new method trains deep networks by separating weight locations from values.
New method shows random, diverse initializations are not essential for deep neural networks.
Mimetic initialization improves Transformer training on small datasets.
The performance of the Self-Organizing Map (SOM) algorithm is dependent on the initial weights of the map. The different initialization methods can broadly be classified into random and data analysis based initialization approach. In this paper, the performance of random initialization (RI) approach is compared to that…
Paper proposes a new strategy to improve initial performance of federated models.
In this paper, we present a novel approach for initializing deep neural networks, i.e., by turning PCA into neural layers. Usually, the initialization of the weights of a deep neural network is done in one of the three following ways: 1) with random values, 2) layer-wise, usually as Deep Belief Network or as auto-encod…
The performance of gradient-based optimization strategies depends heavily on the initial weights of the parametric model. Recent works show that there exist weight initializations from which optimization procedures can find the task-specific parameters faster than from uniformly random initializations and that such a w…
Recent pruning methods at initialization fall short of random pruning's accuracy.
Based on the property that solving the system of linear matrix equations via the column space and the row space projections boils down to an approximation in the least squares error sense, a formulation for learning the weight matrices of the multilayer network can be derived. By exploiting into the vast number of feas…
Next generation deep neural networks for classification hosted on embedded platforms will rely on fast, efficient, and accurate learning algorithms. Initialization of weights in learning networks has a great impact on the classification accuracy. In this paper we focus on deriving good initial weights by modeling the e…
Standard practice in training neural networks involves initializing the weights in an independent fashion. The results of recent work suggest that feature "diversity" at initialization plays an important role in training the network. However, other initialization schemes with reduced feature diversity have also been sh…
A new meta-learning framework that assigns weights to source tasks based on target samples.
New methods train neural networks without changing weights, achieving similar or higher performance.
This paper extends the Gaussian process interpretation of deep networks to more varied weight distributions.
The paper studies neural networks' convergence near origin and saddle points.
The selection of initial parameter values for gradient-based optimization of deep neural networks is one of the most impactful hyperparameter choices in deep learning systems, affecting both convergence times and model performance. Yet despite significant empirical and theoretical analysis, relatively little has been p…
A new method to prune neural networks with iterative randomization improves efficiency.
We define and make initial study of Lie groupoids equipped with a compatible homogeneity (or graded bundle) structure, such objects we will refer to as weighted Lie groupoids. One can think of weighted Lie groupoids as graded manifolds in the category of Lie groupoids. This is a very rich geometrical theory with numero…
A new neural network initialization method is proposed for faster and more accurate training.
Gradient descent converges linearly for neural networks with specific conditions.
Orthogonal initialization does not speed up training in ultra-wide neural networks.
Initializing the weights and the biases is a key part of the training process of a neural network. Unlike the subsequent optimization phase, however, the initialization phase has gained only limited attention in the literature. In this paper we discuss some consequences of commonly used initialization strategies for va…
Deep learning relies on good initialization schemes and hyperparameter choices prior to training a neural network. Random weight initializations induce random network ensembles, which give rise to the trainability, training speed, and sometimes also generalization ability of an instance. In addition, such ensembles pro…
S-GAI initializes MLPs using spectral geometry from data, improving performance.
Deep networks with orthogonal weights show stable fluctuations, improving generalization and training speed.
Understanding how neural networks learn remains one of the central challenges in machine learning research. From random at the start of training, the weights of a neural network evolve in such a way as to be able to perform a variety of tasks, like classifying images. Here we study the emergence of structure in the wei…
Convolutional networks struggle to learn Game of Life, even with lottery ticket weights.
Neural nets trained with linear discriminant initialization converge faster and more accurately.
It is well known that the initialization of weights in deep neural networks can have a dramatic impact on learning speed. For example, ensuring the mean squared singular value of a network's input-output Jacobian is is essential for avoiding the exponential vanishing or explosion of gradients. The stronger condi…
Paper optimizes neural network initialization using SMT solvers.
Generative model initializes 2-layer network weights for small datasets.
Deep neural networks (DNNs) form the backbone of almost every state-of-the-art technique in the fields such as computer vision, speech processing, and text analysis. The recent advances in computational technology have made the use of DNNs more practical. Despite the overwhelming performances by DNN and the advances in…
Wide neural networks become linear, but adding bottlenecks makes them bilinear or multilinear.
Our work presents extensive empirical evidence that layer rotation, i.e. the evolution across training of the cosine distance between each layer's weight vector and its initialization, constitutes an impressively consistent indicator of generalization performance. In particular, larger cosine distances between final an…
This work investigates the ways in which deep learning methods can benefit from random projection (RP), a classic linear dimensionality reduction method. We focus on two areas where, as we have found, employing RP techniques can improve deep models: training neural networks on high-dimensional data and initialization o…
Re-initializing neural networks improves generalization but not as much as other techniques.
An interesting approach to analyzing neural networks that has received renewed attention is to examine the equivalent kernel of the neural network. This is based on the fact that a fully connected feedforward network with one hidden layer, a certain weight distribution, an activation function, and an infinite number of…
Single layer Feedforward Neural Network(FNN) is used many a time as a last layer in models such as seq2seq or could be a simple RNN network. The importance of such layer is to transform the output to our required dimensions. When it comes to weights and biases initialization, there is no such specific technique that co…
New initialization schemes preserve fractional moments of weights in deep networks, improving training and test performance.