The performance of the Self-Organizing Map (SOM) algorithm is dependent on the initial weights of the map. The different initialization methods can broadly be classified into random and data analysis based initialization approach. In this paper, the performance of random initialization (RI) approach is compared to that…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Initial Performance”
New distributed clustering algorithms show resilience to initialization issues.
problem Resilience of distributed gradient-based clustering algorithms to center initialization.
method Distributed gradient-based clustering algorithms with novel center initialization.
result The algorithms are more resilient to initialization compared to baseline methods.
Paper optimizes neural network initialization using SMT solvers.
problem Improving neural network performance through better initialization.
method Reduces initialization to SMT problem solving.
result Proposed method achieves better performance than random initialization.
Study shows different initialization schemes for LoRA finetuning impact performance.
problem The impact of initialization schemes on LoRA finetuning performance.
method Compared two initialization schemes: B=0, A=random vs. A=0, B=random.
result First initialization scheme yields better performance on average.
In this paper, we firstly give a brief introduction of expectation maximization (EM) algorithm, and then discuss the initial value sensitivity of expectation maximization algorithm. Subsequently, we give a short proof of EM's convergence. Then, we implement experiments with the expectation maximization algorithm (We im…
Improved portfolio optimization reduces sensitivity to neural network initialization.
problem High sensitivity to neural network initialization in portfolio optimization.
method Robust end-to-end framework for risk budgeting portfolios.
result Enhanced stability in portfolio optimization without compromising performance.
Effective theory for Transformer initialization improves model performance.
problem Improving performance of Transformers at initialization.
method Effective-theory analysis of signal propagation in wide and deep Transformers.
result Particular width scalings of initialization and training hyperparameters.
The success of lottery ticket initializations (Frankle and Carbin, 2019) suggests that small, sparsified networks can be trained so long as the network is initialized appropriately. Unfortunately, finding these "winning ticket" initializations is computationally expensive. One potential solution is to reuse the same wi…
Improves LSTM performance by initializing states via manifold learning.
problem Improving LSTM performance through better initialization.
method Learning an intrinsic data manifold to initialize LSTM internal states.
result Improved LSTM performance through consistent initialization.
Deep neural networks achieve state-of-the-art performance for a range of classification and inference tasks. However, the use of stochastic gradient descent combined with the nonconvexity of the underlying optimization problems renders parameter learning susceptible to initialization. To address this issue, a variety o…
Graph neural networks benefit from a new initialization method that improves node learning.
problem Poor initialization in GNNs leads to slower convergence and increased training instability.
method Integrates a statistically grounded one-hot graph encoder embedding (GEE) into standard GNNs.
result GG framework provides consistent and substantial performance gains in node classification.
Residual networks (ResNet) and weight normalization play an important role in various deep learning applications. However, parameter initialization strategies have not been studied previously for weight normalized networks and, in practice, initialization methods designed for un-normalized networks are used as a proxy.…
Paper proposes a new strategy to improve initial performance of federated models.
problem Weight divergence in Federated Averaging (FedAvg) leads to poor initial performance in federated models.
method Local continual training with importance weights evaluated on a proxy dataset.
result The method significantly improves the initial performance of federated models with minimal extra communication costs.
Improved performance of factorized neural layers through spectral initialization and Frobenius decay.
problem Improving the performance of factorized neural layers in various deep learning contexts.
method Spectral initialization and Frobenius decay for initialization and regularization.
result Spectral initialization and Frobenius decay lead to improved performance across multiple deep learning settings.
New CNN initialization scheme derived from modern architectures.
problem Stability of CNN model parameters initialization.
method Derived new initialization scheme from modern CNN architectures.
result New initialization method outperforms de facto standard schemes.
New method improves meta-learning performance by task-specific initialization.
problem Difficulties in generalizing and achieving theoretical guarantees in conditional meta-learning.
method Structured prediction approach for task-specific initialization.
result TASML improves performance of existing meta-learning models.
New method learns good initialization for gradient descent from past solutions.
problem Challenges in solving non-convex optimization problems.
method Learning good initialization rules from previous solutions.
result Our approach performs better than random initialization in various non-convex problems.
Meta-learning improves GNN initializations for low-resource drug discovery.
problem Limited labeled data hinders deep learning in drug discovery.
method Model-Agnostic Meta-Learning (MAML) and its variants for graph neural networks initializations.
result Meta-initializations outperform multi-task pre-training baselines on 16 out of 20 tasks and all out-of-distribution tasks.
Combines PCA and AMP for better signal estimation in noisy data.
problem Estimating a rank-1 signal in rotationally invariant noise.
method Combines PCA and AMP, with PCA initialization at the start of AMP.
result Rigorous asymptotic characterization of the new estimator's performance.
New initialization schemes preserve fractional moments of weights in deep networks, improving training and test performance.
problem Heavy-tailed distribution of stochastic gradients in DNNs during training.
method Developed initialization schemes that preserve any given fractional moment of order s < 2 over layers for various activations.
result The network output admits a heavy-tailed distribution with finite moments, improving training and test performance.
Improved LLM pre-training performance through better weight and variance control.
problem Improper weight and variance control in LLM pre-training affects downstream task performance.
method Introduced Layer Index Rescaling (LIR) and Target Variance Rescaling (TVR) techniques.
result Substantial improvements in downstream task performance (up to 4.6%) and reduced extreme activation values.
The selection of initial parameter values for gradient-based optimization of deep neural networks is one of the most impactful hyperparameter choices in deep learning systems, affecting both convergence times and model performance. Yet despite significant empirical and theoretical analysis, relatively little has been p…
GD with large init shows incremental learning in matrix factorization.
problem Understanding GD's behavior with large initial values in matrix factorization.
method Signal-to-noise ratio concepts and inductive arguments.
result Uncovering an incremental learning phenomenon in GD with large initialization.
Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial Statescs.LG
This paper studies how gradient descent in control systems can perform well on unseen data.
problem The extent of a learned controller's ability to extrapolate to unseen initial states.
method Theoretical study of policy gradient in Linear Quadratic Regulator (LQR) problems, focusing on the role of exploration.
result The performance of a learned controller on unseen initial states depends on the degree of exploration induced by the system.
Deep neural networks (DNNs) form the backbone of almost every state-of-the-art technique in the fields such as computer vision, speech processing, and text analysis. The recent advances in computational technology have made the use of DNNs more practical. Despite the overwhelming performances by DNN and the advances in…
AutoInit automatically finds good neural network initialization.
problem Finding optimal neural network initialization is crucial but time-consuming.
method Uses Jacobian tuning to automatically adjust network hyperparameters.
result The method finds good initialization for various network architectures.
Changing initialization scale affects deep model generalization, leading to memorization or improved performance.
problem Understanding how initialization scale impacts deep model generalization and memorization.
method Experimental setup with varying initialization scales, analysis of activation and loss functions, and development of an alignment measure.
result Increasing initialization scale leads to memorization, and decreasing it improves generalization, depending on activation and loss functions.
In this paper, we investigate the mean curvature flow having equifocal submanifolds as initial data. The investigation are performed by investigating the mean curvature flow having the lifted submanifolds to a Hilbert space through a Riemannian submersion as initial data.
Randomly initialized networks can perform as well as pruned networks.
problem Understanding and improving network pruning methods.
method Sanity checks on recent pruning methods, proposing random tickets.
result Randomly initialized networks can perform as well as pruned networks.
Paper proposes AMP with spectral initialization for robust signal estimation.
problem Signal estimation from generalized linear model measurements with correlated initialization.
method Approximate message passing (AMP) with spectral initialization.
result Characterization of AMP with spectral initialization in high-dimensional limit.
A new design methodology for neural networks that is guided by traditional algorithm design is presented. To prove our point, we present two heuristics and demonstrate an algorithmic technique for incorporating additional weights in their signal-flow graphs. We show that with training the performance of these networks …
S-GAI initializes MLPs using spectral geometry from data, improving performance.
problem Lack of guidance on initial weights encoding data geometry.
method S-GAI uses SVD to estimate spectral class geometry, initializing MLPs from training data.
result S-GAI-initialized MLPs start from a more informative hidden state and achieve comparable accuracy.
Bayesian method improves neural net convergence for character recognition.
problem Improving convergence rate of neural network training algorithms.
method Customization of Kalman filter into Bayesian statistics for initialization of weights.
result Improved convergence rate for backpropagation training algorithm.
Initialization of parameters in deep neural networks has been shown to have a big impact on the performance of the networks (Mishkin & Matas, 2015). The initialization scheme devised by He et al, allowed convolution activations to carry a constrained mean which allowed deep networks to be trained effectively (He et al.…
Gradient Descent (GD) approximators often fail in the solution space with multiple scales of convexities, i.e., in subspace learning and neural network scenarios. To handle that, one solution is to run GD multiple times from different randomized initial states and select the best solution over all experiments. However,…
Proposes a method to optimize neural network initialization using marginal likelihood maximization.
problem Optimizing hyperparameters for neural network initialization.
method Leverages the connection between neural networks and Gaussian processes to infer optimal hyperparameters.
result Marginal likelihood maximization provides near-optimal prediction performance on MNIST classification tasks.
Re-initializing neural networks improves generalization but not as much as other techniques.
problem Understanding when and how re-initialization improves neural network performance.
method Empirical comparison of re-initialization with standard training and various regularization techniques.
result Re-initialization is beneficial for generalization but not as much as other techniques, especially when combined with careful tuning of hyperparameters.
Deep networks retain initial bias after training, affecting generalization.
problem Understanding how much initial bias in neural networks survives training.
method Introduced initialization memory to measure initial bias's survival.
result SGD can preserve initial bias, while Adam-family methods erase it.
Max-Pooling Loss Training of Long Short-Term Memory Networks for Small-Footprint Keyword Spottingcs.CL
We propose a max-pooling based loss function for training Long Short-Term Memory (LSTM) networks for small-footprint keyword spotting (KWS), with low CPU, memory, and latency requirements. The max-pooling loss training can be further guided by initializing with a cross-entropy loss trained network. A posterior smoothin…
FreSh shifts model's initial frequency spectrum to match target signal, improving neural representation performance.
problem MLPs' low-frequency bias limits capturing high-frequency details accurately.
method FreSh selects embedding hyperparameters to align model's initial output spectrum with target signal's spectrum.
result FreSh improves performance across various neural representation methods and tasks with minimal computational overhead.
Study finds dividend policy has no significant effect on IPO stock prices.
problem Impact of dividend policy on IPO price performance.
method Long-run performance statistics and GARCH model, dummy variable used.
result Dividend policy has no significant effect on IPO stock prices.
Neural Turing Machines (NTMs) are an instance of Memory Augmented Neural Networks, a new class of recurrent neural networks which decouple computation from memory by introducing an external memory unit. NTMs have demonstrated superior performance over Long Short-Term Memory Cells in several sequence learning tasks. A n…
Based on the property that solving the system of linear matrix equations via the column space and the row space projections boils down to an approximation in the least squares error sense, a formulation for learning the weight matrices of the multilayer network can be derived. By exploiting into the vast number of feas…
US firms improve ESG performance in response to China trade shock.
problem Impact of China trade shock on US ESG performance.
method Trade policy experiment exploiting tariff changes.
result Greater import competition from China increases US firm ESG performance.
HOPE improves SSMs for long-memory tasks with robust initialization and training.
problem Improving state-space models for long-memory tasks with robust initialization and training.
method Developed a new parameterization scheme called HOPE using Hankel operators and Markov parameters.
result HOPE improves SSMs' performance on Long-Range Arena tasks and demonstrates non-decaying memory.
Mixed datasets consist of both numeric and categorical attributes. Various k-means-based clustering algorithms have been developed for these datasets. Generally, these algorithms use random partition as a starting point, which tends to produce different clustering results for different runs. In this paper, we propose, …
Optimal Condition for Initialization Variance in Deep Neural Networks: An SGD Dynamics Perspectivestat.ML
Optimizes deep neural network initialization variance for better performance.
problem Improving deep neural network performance through optimal initialization variance.
method Using SGD dynamics and Fokker-Planck equations, we study the relationship between initialization and expected loss function.
result An optimal condition for initialization variance that leads to lower training loss and higher test accuracy.
GNNs with random node initialization are shown to be universally expressive.
problem Limitations of standard GNNs in distinguishing graphs.
method Random node initialization (RNI) to enhance GNNs' expressive power.
result GNNs with RNI are proven to be universally expressive.