Method measures weight similarity in neural networks using normalization and statistical inference.
problem Quantifying weight similarity in non-convex neural networks.
method Chain normalization rule and hypothesis-training-testing statistical inference.
result Weights of identical neural networks converge to similar local solutions.
The paper predicts edge weights in weighted directed networks using metric geometry.
problem Predicting edge weights in weighted directed networks.
method Introducing new types of weighted directed networks (AWDNs), constructing metrics, and proposing modified kNN and SVM methods.
result The proposed methods outperform traditional approaches in predicting edge weights.
Adversarial robustness improved by sparsity in network weights.
problem How sparsity affects adversarial robustness in neural networks.
method Theoretical proof and experimental validation of adversarial pruning methods.
result Weights sparsity improves adversarial robustness, especially through inheritance from smaller networks.
Formula found for neural network error with fixed weights.
problem Understanding error in neural networks with fixed weights.
method Provided an explicit formula for approximation error.
result Explicit formula for neural network error with fixed weights.
DeepWeightFlow generates diverse neural network weights efficiently.
problem Generating complete neural network weights efficiently and accurately.
method Flow Matching in weight space with Git Re-Basin and TransFusion.
result DeepWeightFlow generates high-accuracy neural networks without fine-tuning.
Hardness proven for neural networks with natural weights.
problem Difficulty in learning neural networks with weights from natural distributions.
method Proved hardness for depth-2 networks with natural weights distributions.
result Most networks are hard to learn with natural weights.
A new model for detecting overlapping communities in weighted networks.
problem Community detection in overlapping weighted networks with mixed membership and edge weights.
method Mixed membership distribution-free (MMDF) model with an efficient spectral algorithm and fuzzy weighted modularity.
result The MMDF model can estimate community memberships and evaluate community quality for weighted networks.
Unified framework for measuring concentration in weighted networks considering both weight distributions and network structure.
problem Traditional indices neglect the topology of relationships among network elements.
method Develops a family of topology-aware concentration indices that jointly account for weight distributions and network structure.
result The proposed indices preserve key properties and allow concentration to be evaluated across different dimensions of dependence.
Proposes volumization for neural networks to control bias-variance tradeoff.
problem Improving generalization and preventing memorization in neural networks.
method Defines a physical volume for weights, interpolating between L2 and L∞ regularization.
result Volumization interpolates between weight decay and clipping, improving generalization.
New algorithm detects communities in weighted networks, improving on binary ones.
problem Few methods exist for detecting communities in weighted networks.
method Pseudo-likelihood approach for weighted stochastic block model.
result The method is consistent and works well for both homogeneous and heterogeneous networks.
Bayesian priors for neural networks are improved by incorporating weight correlations and tail behavior.
problem Improving Bayesian priors for neural networks to better reflect true beliefs and performance.
method Analyzed summary statistics of neural network weights in different architectures and incorporated these observations into new priors.
result Improved performance on image classification datasets by using new priors that account for weight correlations and tail behavior.
Not all neural network architectures are created equal, some perform much better than others for certain tasks. But how important are the weight parameters of a neural network compared to its architecture? In this work, we question to what extent neural network architectures alone, without learning any weight parameter…
This paper proposes a discrimination technique for vertices in a weighted network. We assume that the edge weights and adjacencies in the network are conditionally independent and that both sources of information encode class membership information. In particular, we introduce a edge weight distribution matrix to the s…
New bound for neural networks with full-rank weights, independent of network width.
problem Understanding generalization of neural networks with full-rank weight matrices.
method Using Koopman operators to derive a tighter generalization bound for full-rank weight matrices.
result The bound is tighter than existing norm-based bounds when condition numbers are small.
SGD and weight decay encourage neural networks to learn low-rank weight matrices.
problem The bias of SGD towards low-rank weight matrices in neural networks.
method The study investigates the effect of SGD and weight decay on the rank of weight matrices in neural networks, both theoretically and empirically.
result Training with SGD and weight decay induces a bias towards rank minimization in weight matrices, which becomes more pronounced with smaller batch sizes and stronger weight decay.
New regularizer improves neural network robustness and generalization.
problem Ineffective weight decay for networks with homogeneous activation functions.
method Proposes an invariant regularizer to penalize intrinsic weight norms.
result Improves generalization and adversarial robustness on various datasets.
This study uncovers how neural architectures and weights interact in classification tasks.
problem Understanding the role of neural architecture and weights in classification performance.
method Developed a novel method to find optimal task-specific architectures as binary networks with {0, 1}-valued weights, using approximate gradient descent.
result Well-trained architectures may not require fine-tuning of weights, highlighting the importance of structure over weights.
A new method trains deep networks by separating weight locations from values.
problem Training deep networks efficiently and effectively.
method Lookahead Permutation (LaPerm) to train DNNs by reconnecting weights.
result LaPerm can train DNNs with random and dense, sparse, or single-valued initial weights.
This paper examines weight initialization for 1-Lipschitz networks to improve robustness against adversarial attacks.
problem Improving the robustness of deep neural networks against adversarial attacks.
method Examined weight parametrization of AOL and SLL networks, calculated weight variance bounds, and demonstrated weight decay.
result Weight initialization causes deep 1-Lipschitz networks to decay to zero, and weight variance does not affect output variance distribution.
Complex-valued neural networks avoid spurious local minima.
problem Finding spurious local minima in neural networks.
method Proved no spurious local minima for shallow complex neural networks with quadratic activations.
result Complex-valued weights eliminate spurious local minima in neural networks.
Neural networks with integer weights approximate continuous functions efficiently.
problem Approximating continuous functions using neural networks with integer weights.
method Integrates superexpressive activation functions and integer weights.
result Convergence rate of order n2β+d−2βlog2n for neural network regression. RSO uses random weight perturbations to train deep networks without gradients.
problem Training deep neural networks efficiently and without gradient information.
method RSO is a gradient-free Markov Chain Monte Carlo approach that updates weights based on mini-batch loss reduction.
result RSO achieves high accuracy (99.1% on MNIST) with significantly fewer updates than traditional methods.
This paper introduces hierarchical Gaussian process priors for neural networks to capture weight correlations and inductive biases.
problem Capturing weight correlations and inductive biases in neural networks.
method Hierarchical Gaussian process priors with unit embeddings and input-dependent kernels.
result Hierarchical Gaussian process priors provide competitive predictive performance and desirable uncertainty estimates.
Predicting neural network accuracy from weights without testing.
problem Predicting neural network performance based on weights alone.
method Used simple statistics of weights to rank neural networks' performance.
result Simple predictors can rank networks' performance with high accuracy (R2 score > 0.98).
Improved neural networks by averaging late-stage weights.
problem Improving the performance of neural networks.
method Ensemble late-stage weights and average them.
result Augmenting standard models with late-phase weights improves generalization.
Adaptive weights improve physics-informed neural networks and deep operator networks.
problem Training physics-informed neural networks and deep operator networks can be challenging, leading to unsatisfactory accuracy and efficiency.
method Proposes a pointwise adaptive weighting method that balances the residual decay rate across different training points.
result Our proposed approach of balanced residual decay rates offers advantages including bounded weights, high prediction accuracy, fast convergence rate, low training uncertainty, low computational cost, and ease of hyperparameter tuning.
Infinite CNNs lose spatial correlations, but can be restored by correlated weights.
problem Infinite CNNs lose spatial correlations, which are crucial for their performance.
method Introduced correlated weights to restore spatial correlations in infinite CNNs.
result Optimal performance is achieved with a moderate level of weight correlation.
New neural network rates for unbounded domains with weighted Sobolev spaces.
problem Improving neural network approximation rates for unbounded domains.
method Embedding results for weighted Fourier-Lebesgue spaces in weighted Sobolev spaces, followed by asymptotic approximation rates.
result Asymptotic approximation rates for shallow neural networks without curse of dimensionality for unbounded domains and Muckenhoupt weights.
DFM model detects communities in weighted networks without distributional assumptions.
problem Challenges in community detection for weighted networks.
method Distribution-Free Model (DFM) for weighted networks, using spectral clustering.
result Spectral clustering yields consistent community detection under DFM.
Echo state networks with random weights can approximate any continuous system.
problem Approximating continuous dynamical systems using echo state networks.
method Randomly generated internal weights and a sampling procedure for activation functions.
result Echo state networks with random weights can approximate any continuous casual time-invariant operators with high probability.
Neural networks with learned biases can approximate any function.
problem Whether neural networks with only learned biases can approximate any continuous function.
method Theoretical and numerical analysis of random weights and learned biases in neural networks.
result Feedforward and recurrent neural networks with random weights can approximate any continuous function and dynamical systems.
New model for detecting communities in weighted bipartite networks.
problem Lack of models for weighted bipartite networks.
method Introducing Bipartite Distribution-Free model and its extension.
result Spectral algorithms for consistent estimation of node labels.
Two retraining techniques outperform fine-tuning in neural network pruning.
problem Improving accuracy and compression in neural network pruning.
method Weight rewinding and learning rate rewinding compared to fine-tuning.
result Rewinding techniques outperform fine-tuning in accuracy and compression.
New method clusters weighted directed networks using motifs.
problem Clustering directed networks fails to consider higher-order structure and edge weights.
method Motif-based weighted spectral clustering with new matrix formulae.
result Scalable and effective clustering on large graphs and real-world data.
With the development of deep neural networks, the size of network models becomes larger and larger. Model compression has become an urgent need for deploying these network models to mobile or embedded devices. Model quantization is a representative model compression technique. Although a lot of quantization methods hav…
Reintroduces straight-through estimators for binary neural networks.
problem Training neural networks with binary weights and activations is challenging due to gradient issues and discrete weight optimization.
method Derives ST methods as estimators in the SBN model, analyzes properties and estimation accuracy, explains latent weights and mirror descent method.
result Reintroduces ST methods as sound approximations and provides clearer application and improvements.
New methods train neural networks without changing weights, achieving similar or higher performance.
problem Training neural networks efficiently with randomly initialized weights.
method Switching connections on and off, flipping weights' signs, minimizing changed connections.
result Achieves similar or higher performance with less computational cost than training all weights.
Residual Network (ResNet) is the state-of-the-art architecture that realizes successful training of really deep neural network. It is also known that good weight initialization of neural network avoids problem of vanishing/exploding gradients. In this paper, simplified models of ResNets are analyzed. We argue that good…
Stochastic variational inference for Bayesian deep neural network (DNN) requires specifying priors and approximate posterior distributions over neural network weights. Specifying meaningful weight priors is a challenging problem, particularly for scaling variational inference to deeper architectures involving high dime…
LAWN normalizes logits to improve deep network adaptability and generalization.
problem Large logits and weights lead to overfitting in deep networks.
method Logit Attenuating Weight Normalization (LAWN) constrains weight norms in the final sub-network.
result LAWN improves generalization and adaptability of deep networks.
Bayesian weight priors improve neural network learning of identity relations.
problem Neural networks struggle to learn abstract and systematic relations, especially identity relations.
method Extended RBP approach using Bayesian weight priors as a regularization term.
result Bayesian weight priors lead to perfect generalization for identity relations and do not hinder standard neural network learning.
Paper develops a framework to optimize neural networks using weighted metrics.
problem Discrepancy between maximizing weighted classification scores and minimizing loss function.
method Formalizes weighted classification metrics and constructs corresponding losses.
result Framework includes well-established approaches like cost-sensitive learning and weighted cross entropy.
BiTAT improves neural network quantization for edge devices by focusing on weight dependencies and disentangling them.
problem Performance degradation of compact neural networks under extreme quantization.
method Task-dependent Aggregated Transformation (BiTAT) method that orthonormalizes weights and progressively quantizes them.
result BiTAT effectively preserves model performance on ImageNet and CIFAR-100 with compact backbones.
Partial fusion combines neural networks to balance accuracy and efficiency.
problem Balancing accuracy and computational cost in neural networks.
method Extending weight aggregation methods based on neuron-level similarity, using partial optimal transport to match similar neurons.
result Achieves a flexible tradeoff between computational cost and performance.
Recently, in the paper "Weight Agnostic Neural Networks" Gaier & Ha utilized architecture search to find networks where the topology completely encodes the knowledge. However, architecture search in topology space is expensive. We use the existing framework of binarized networks to find performant topologies by constra…
Neural networks' weights don't converge to stationary points but training loss stabilizes.
problem The disconnect between theoretical analyses and neural network training practice.
method An invariant measure perspective inspired by ergodic theory of dynamical systems.
result The distribution of weights converges to an approximate invariant measure, explaining loss stabilization.
Analyzes minima of deep linear networks with weight decay.
problem Understanding the loss landscape of deep neural networks.
method Analytical solutions for global minima with weight decay and stochastic neurons.
result The origin is a special point with qualitatively different minima in networks with more than 1 hidden layer.
New framework for dense weighted networks with community-specific patterns.
problem Dense networks with varying edge weights across communities.
method Proposes a new model with functions mapping node characteristics to edge weights, requiring few parameters.
result Developed a bootstrap methodology for generating new networks.