Method measures weight similarity in neural networks using normalization and statistical inference.
problem Quantifying weight similarity in non-convex neural networks.
method Chain normalization rule and hypothesis-training-testing statistical inference.
result Weights of identical neural networks converge to similar local solutions.
Formula found for neural network error with fixed weights.
problem Understanding error in neural networks with fixed weights.
method Provided an explicit formula for approximation error.
result Explicit formula for neural network error with fixed weights.
DeepWeightFlow generates diverse neural network weights efficiently.
problem Generating complete neural network weights efficiently and accurately.
method Flow Matching in weight space with Git Re-Basin and TransFusion.
result DeepWeightFlow generates high-accuracy neural networks without fine-tuning.
Proposes volumization for neural networks to control bias-variance tradeoff.
problem Improving generalization and preventing memorization in neural networks.
method Defines a physical volume for weights, interpolating between L2 and L∞ regularization.
result Volumization interpolates between weight decay and clipping, improving generalization.
Bayesian priors for neural networks are improved by incorporating weight correlations and tail behavior.
problem Improving Bayesian priors for neural networks to better reflect true beliefs and performance.
method Analyzed summary statistics of neural network weights in different architectures and incorporated these observations into new priors.
result Improved performance on image classification datasets by using new priors that account for weight correlations and tail behavior.
Complex-valued neural networks avoid spurious local minima.
problem Finding spurious local minima in neural networks.
method Proved no spurious local minima for shallow complex neural networks with quadratic activations.
result Complex-valued weights eliminate spurious local minima in neural networks.
New learning process for neural classifiers simplifies decision weights.
problem Training decision layer weights in neural classifiers.
method Choosing appropriate loss functions and solving constrained optimization problems.
result A new learning process for pre-decision weights that is simple and effective.
Not all neural network architectures are created equal, some perform much better than others for certain tasks. But how important are the weight parameters of a neural network compared to its architecture? In this work, we question to what extent neural network architectures alone, without learning any weight parameter…
SGD and weight decay encourage neural networks to learn low-rank weight matrices.
problem The bias of SGD towards low-rank weight matrices in neural networks.
method The study investigates the effect of SGD and weight decay on the rank of weight matrices in neural networks, both theoretically and empirically.
result Training with SGD and weight decay induces a bias towards rank minimization in weight matrices, which becomes more pronounced with smaller batch sizes and stronger weight decay.
Improved neural networks by averaging late-stage weights.
problem Improving the performance of neural networks.
method Ensemble late-stage weights and average them.
result Augmenting standard models with late-phase weights improves generalization.
Neural networks with learned biases can approximate any function.
problem Whether neural networks with only learned biases can approximate any continuous function.
method Theoretical and numerical analysis of random weights and learned biases in neural networks.
result Feedforward and recurrent neural networks with random weights can approximate any continuous function and dynamical systems.
Neural networks with integer weights approximate continuous functions efficiently.
problem Approximating continuous functions using neural networks with integer weights.
method Integrates superexpressive activation functions and integer weights.
result Convergence rate of order n2β+d−2βlog2n for neural network regression. RSO uses random weight perturbations to train deep networks without gradients.
problem Training deep neural networks efficiently and without gradient information.
method RSO is a gradient-free Markov Chain Monte Carlo approach that updates weights based on mini-batch loss reduction.
result RSO achieves high accuracy (99.1% on MNIST) with significantly fewer updates than traditional methods.
Predicting neural network accuracy from weights without testing.
problem Predicting neural network performance based on weights alone.
method Used simple statistics of weights to rank neural networks' performance.
result Simple predictors can rank networks' performance with high accuracy (R2 score > 0.98).
New bound for neural networks with full-rank weights, independent of network width.
problem Understanding generalization of neural networks with full-rank weight matrices.
method Using Koopman operators to derive a tighter generalization bound for full-rank weight matrices.
result The bound is tighter than existing norm-based bounds when condition numbers are small.
This study uncovers how neural architectures and weights interact in classification tasks.
problem Understanding the role of neural architecture and weights in classification performance.
method Developed a novel method to find optimal task-specific architectures as binary networks with {0, 1}-valued weights, using approximate gradient descent.
result Well-trained architectures may not require fine-tuning of weights, highlighting the importance of structure over weights.
The study examines Fisher information matrices and neural tangent kernels for simple ReLU networks with random weights.
problem Understanding the relationship between Fisher information matrices and neural tangent kernels for 2-layer ReLU networks.
method Analyzes Fisher information matrices and neural tangent kernels for 2-layer ReLU networks with random hidden weights, focusing on spectral decomposition and eigenfunctions.
result Obtained an approximation formula for functions represented by 2-layer neural networks.
Bayesian method improves neural net convergence for character recognition.
problem Improving convergence rate of neural network training algorithms.
method Customization of Kalman filter into Bayesian statistics for initialization of weights.
result Improved convergence rate for backpropagation training algorithm.
Stochastic variational inference for Bayesian deep neural network (DNN) requires specifying priors and approximate posterior distributions over neural network weights. Specifying meaningful weight priors is a challenging problem, particularly for scaling variational inference to deeper architectures involving high dime…
This paper introduces hierarchical Gaussian process priors for neural networks to capture weight correlations and inductive biases.
problem Capturing weight correlations and inductive biases in neural networks.
method Hierarchical Gaussian process priors with unit embeddings and input-dependent kernels.
result Hierarchical Gaussian process priors provide competitive predictive performance and desirable uncertainty estimates.
Neural networks' weights don't converge to stationary points but training loss stabilizes.
problem The disconnect between theoretical analyses and neural network training practice.
method An invariant measure perspective inspired by ergodic theory of dynamical systems.
result The distribution of weights converges to an approximate invariant measure, explaining loss stabilization.
Paper develops a framework to optimize neural networks using weighted metrics.
problem Discrepancy between maximizing weighted classification scores and minimizing loss function.
method Formalizes weighted classification metrics and constructs corresponding losses.
result Framework includes well-established approaches like cost-sensitive learning and weighted cross entropy.
BiTAT improves neural network quantization for edge devices by focusing on weight dependencies and disentangling them.
problem Performance degradation of compact neural networks under extreme quantization.
method Task-dependent Aggregated Transformation (BiTAT) method that orthonormalizes weights and progressively quantizes them.
result BiTAT effectively preserves model performance on ImageNet and CIFAR-100 with compact backbones.
Paper proves multiplicative weight updates can train neural networks without learning rate tuning.
problem Vanishing and exploding gradients in gradient descent for compositional functions.
method Proves descent lemma for compositional functions using multiplicative weight updates and derives Madam optimizer.
result Madam optimizer trains state-of-the-art neural networks without learning rate tuning.
This paper presents a supervised learning algorithm, namely, the Synaptic Efficacy Function with Meta-neuron based learning algorithm (SEF-M) for a spiking neural network with a time-varying weight model. For a given pattern, SEF-M uses the learning algorithm derived from meta-neuron based learning algorithm to determi…
New neural network rates for unbounded domains with weighted Sobolev spaces.
problem Improving neural network approximation rates for unbounded domains.
method Embedding results for weighted Fourier-Lebesgue spaces in weighted Sobolev spaces, followed by asymptotic approximation rates.
result Asymptotic approximation rates for shallow neural networks without curse of dimensionality for unbounded domains and Muckenhoupt weights.
This paper investigates weight-sharing in NAS, revealing its impact and providing solutions.
problem Reducing the time and computational cost of training neural networks.
method Comprehensive experiments on weight-sharing in NAS, analyzing variance and interference.
result Properly reducing weight sharing can reduce variance and improve model performance.
Weight Decay induces low-rank weight matrices in neural networks, improving generalization.
problem Improving generalization in neural networks.
method Training ReLU NN with Weight Decay and Stochastic Gradient Descent.
result The weight matrix of a trained NN is approximately rank-two.
New methods train neural networks without changing weights, achieving similar or higher performance.
problem Training neural networks efficiently with randomly initialized weights.
method Switching connections on and off, flipping weights' signs, minimizing changed connections.
result Achieves similar or higher performance with less computational cost than training all weights.
MLDS dataset reveals hidden model behavior via weight-space analysis.
problem Neural networks' opacity makes them hard to evaluate.
method Presented MLDS dataset of trained neural networks.
result Weight-space analysis reveals meaningful divergence with small changes in training data.
Until recently, research on artificial neural networks was largely restricted to systems with only two types of variable: Neural activities that represent the current or recent input and weights that learn to capture regularities among inputs, outputs and payoffs. There is no good reason for this restriction. Synapses …
New regularizer improves neural network robustness and generalization.
problem Ineffective weight decay for networks with homogeneous activation functions.
method Proposes an invariant regularizer to penalize intrinsic weight norms.
result Improves generalization and adversarial robustness on various datasets.
Recently, Neural networks have seen a huge surge in its adoption due to their ability to provide high accuracy on various tasks. On the other hand, the existence of adversarial examples have raised suspicions regarding the generalization capabilities of neural networks. In this work, we focus on the weight matrix learn…
Bayesian neural networks learn weights with closed-form updates.
problem Efficiently learning Bayesian neural networks with closed-form updates.
method Closed-form Bayesian inference for online learning of Gaussian-weighted BNNs.
result Closed-form expressions for sequential/online training of BNNs.
Estimates causal effects using neural networks for balancing covariates.
problem Estimating causal effects from observational data.
method Neural Balancing Weights (NBW) using α-divergence for density ratio estimation. result Generalized approach for balancing multidimensional data.
EWC helps prevent forgetting in neural networks by adjusting weights dynamically.
problem Preventing forgetting in neural networks during training.
method EWC adjusts weights dynamically to prevent forgetting.
result EWC effectively prevents catastrophic forgetting in neural networks.
Hardness proven for neural networks with natural weights.
problem Difficulty in learning neural networks with weights from natural distributions.
method Proved hardness for depth-2 networks with natural weights distributions.
result Most networks are hard to learn with natural weights.
Bayesian weight priors improve neural network learning of identity relations.
problem Neural networks struggle to learn abstract and systematic relations, especially identity relations.
method Extended RBP approach using Bayesian weight priors as a regularization term.
result Bayesian weight priors lead to perfect generalization for identity relations and do not hinder standard neural network learning.
Random weights in GNNs match learned weights in performance.
problem Feature rank collapse in GNNs.
method Replacing learned weights with random weights.
result Random weights achieve comparable performance to learned weights, reducing training time and memory usage.
Infinite CNNs lose spatial correlations, but can be restored by correlated weights.
problem Infinite CNNs lose spatial correlations, which are crucial for their performance.
method Introduced correlated weights to restore spatial correlations in infinite CNNs.
result Optimal performance is achieved with a moderate level of weight correlation.
New duality found linking neural network weights and activities for better generalization.
problem Understanding and improving neural network generalization.
method Activity-weight duality mapping between neural network layers.
result Generalization loss can be decomposed into geometric factors of sharpness and weight standard deviation.
Enhances neural network regression performance by modeling weight and variance uncertainty.
problem Improving predictive performance of neural networks for regression tasks.
method Extended Blundell's framework to include variance uncertainty, using a full posterior distribution over variance parameters.
result Explicitly modeling variance uncertainty improves generalization of Bayesian neural networks.
The paper shows neural networks can approximate functions over non-compact domains with non-polynomial activation.
problem Approximating functions over non-compact domains using neural networks.
method Using single-hidden-layer feedforward neural networks with non-polynomial activation functions over non-compact subsets of Euclidean spaces.
result Neural networks can approximate functions in weighted Ck-spaces and weighted Sobolev spaces over unbounded domains. A new method to improve deep neural networks using weight rescaling.
problem Overfitting and sensitivity to hyperparameters in weight decay.
method Weight rescaling (WRS) to control weight norm and prevent overfitting.
result WRS outperforms weight decay and other methods in various applications.
Neural networks trained with backpropagation, the standard algorithm of deep learning which uses weight transport, are easily fooled by existing gradient-based adversarial attacks. This class of attacks are based on certain small perturbations of the inputs to make networks misclassify them. We show that less biologica…
New learning rules for wide neural networks without backpropagation.
problem Training wide neural networks efficiently and without backpropagation.
method Input-weight alignment driven by gradient descent in the NTK regime.
result Biologically-motivated learning rules equivalent to backpropagation in wide networks.
Understanding how neural networks learn remains one of the central challenges in machine learning research. From random at the start of training, the weights of a neural network evolve in such a way as to be able to perform a variety of tasks, like classifying images. Here we study the emergence of structure in the wei…
Generalizes neural tangent kernel analysis for two-layer networks with noise and regularization.
problem Limitations of NTK analysis in deep learning practice.
method Generalized NTK analysis for two-layer neural networks with weight decay and gradient noise.
result Noisy gradient descent with weight decay exhibits 'kernel-like' behavior and converges linearly.