Law derived for neural networks with sparse connections.
problem Understanding the behavior of neural networks with sparse connections.
method Law of large numbers for empirical distribution of parameters derived.
result Law for neural networks with sparse connections derived.
Geodesics connect model modes in neural network loss landscapes.
problem Connecting modes in neural network loss landscapes.
method Reframed mode connectivity in Information Geometry, hypothesized geodesics as mode-connecting paths, proposed algorithm to approximate geodesics.
result Geodesics achieve mode connectivity in neural networks.
Study analyzes error in ReLU networks with local connections.
problem Improving neural network performance and understanding approximation errors.
method Analyzed approximation error of ReLU networks with local connections.
result Error estimate depends on depth and width of hidden layers.
This paper improves deep neural network approximation for fully connected networks, achieving optimal convergence rates.
problem Improving approximation of fully connected deep neural networks for optimal convergence rates.
method Deriving approximation bounds specifically for a narrower fully connected deep neural network.
result Achieves an optimal rate (up to a logarithmic factor) for fully connected deep neural networks.
We propose Sparse Neural Network architectures that are based on random or structured bipartite graph topologies. Sparse architectures provide compression of the models learned and speed-ups of computations, they can also surpass their unstructured or fully connected counterparts. As we show, even more compact topologi…
Paper analyzes the free energy of CNNs with skip connections in Bayesian learning.
problem Dependency of CNNs with skip connections on the number of parameters.
method Examines the Bayesian free energy of CNNs with and without skip connections.
result The upper bound of free energy of Bayesian CNN with skip connections does not depend on overparametrization.
USN improves neural networks with uniform sparse connectivity.
problem Overfitting and limited scalability in classical neural networks.
method Uniform sparse network (USN) with even and sparse connectivity.
result USN outperforms state-of-the-art sparse network models in accuracy, speed, and robustness.
Previous work has questioned the conditions under which the decision regions of a neural network are connected and further showed the implications of the corresponding theory to the problem of adversarial manipulation of classifiers. It has been proven that for a class of activation functions including leaky ReLU, neur…
The paper reveals surprising star-shaped connectivity in neural networks.
problem Understanding mode connectivity in neural network landscapes.
method Fine-grained analysis of connectivity in overparameterized and finite minima cases.
result Star-shaped connectivity exists in neural network landscapes, suggesting near convexity.
Neural networks can learn optimal auction mechanisms and satisfy mode connectivity.
problem Optimal auction design in complex settings.
method Generalized RochetNet and affine maximizer auctions.
result Neural networks (RochetNet and generalized version) satisfy mode connectivity.
The paper connects neural networks to Mahalanobis distance for interpretability.
problem Lack of interpretability in neural networks.
method Establishes a connection between neural network linear layers and Mahalanobis distance.
result Provides a foundation for more interpretable neural network models.
Graph neural networks suffer from oversmoothing, but adding residual connections helps.
problem Oversmoothing in deep graph neural networks where features become indistinguishable.
method Analyzed asymptotic oversmoothing rates with and without residual connections using the multiplicative ergodic theorem.
result Adding residual connections effectively mitigates or prevents oversmoothing.
PAC-Bayesian bounds show fully connected DNNs with Gaussian priors match minimax rates.
problem Theoretical limits of fully connected deep neural networks with Gaussian priors.
method PAC-Bayesian bounds for fully connected Bayesian DNNs with Gaussian priors.
result PAC-Bayesian bounds match minimax-optimal rates in Besov space for nonparametric regression and binary classification.
The paper proves skip connections help neural networks avoid shallow local minima.
problem Understanding how skip connections affect the loss landscape of deep neural networks.
method Theoretical analysis of the topology of loss landscapes of deep ReLU neural networks with skip connections.
result Skip connections help control the connectedness of sub-level sets, avoiding shallow local minima.
Proposes using mode connectivity to improve adversarial robustness of neural networks.
problem Improving adversarial robustness of deep neural networks.
method Employing mode connectivity in loss landscapes to study adversarial robustness and propose methods for improvement.
result Path connection learned using limited bonafide data can effectively mitigate adversarial effects while maintaining original accuracy.
Artificial Neural Networks form the basis of very powerful learning methods. It has been observed that a naive application of fully connected neural networks to data with many irrelevant variables often leads to overfitting. In an attempt to circumvent this issue, a prior knowledge pertaining to what features are relev…
Model predicts loan default risk using dynamic multilayer graph neural networks.
problem Credit risk assessment through borrower connections.
method Dynamic multilayer graph neural network with attention mechanism.
result Attention mechanism improves model performance.
New method connects neural networks to diagrammatic algebra.
problem Constructing permutation equivariant neural networks.
method Schur-Weyl duality between symmetric group and partition algebra.
result Simple diagrammatic method for calculating weight matrices.
Connections between nodes of fully connected neural networks are usually represented by weight matrices. In this article, functional transfer matrices are introduced as alternatives to the weight matrices: Instead of using real weights, a functional transfer matrix uses real functions with trainable parameters to repre…
New methods train neural networks without changing weights, achieving similar or higher performance.
problem Training neural networks efficiently with randomly initialized weights.
method Switching connections on and off, flipping weights' signs, minimizing changed connections.
result Achieves similar or higher performance with less computational cost than training all weights.
A method of simultaneously optimizing both the structure of neural networks and the connection weights in a single training loop can reduce the enormous computational cost of neural architecture search. We focus on the probabilistic model-based dynamic neural network structure optimization that considers the probabilit…
We show that for neural network functions that have width less or equal to the input dimension all connected components of decision regions are unbounded. The result holds for continuous and strictly monotonic activation functions as well as for the ReLU activation function. This complements recent results on approxima…
Interneurons improve learning in neural networks by accelerating convergence.
problem Rapid adaptation to changing input statistics in neural networks.
method Two mathematically tractable recurrent linear neural networks were compared: one with direct recurrent connections and the other with interneurons that mediate recurrent communication.
result The network with interneurons converges more quickly than the network with direct recurrent connections, scaling logarithmically with initialization spectrum.
New method diagnoses criticality in deep neural networks, improving performance.
problem Improving theoretical understanding and practical initialization of deep neural networks.
method Introducing partial Jacobians and deriving recurrence relations for their norms to analyze criticality.
result Proper stacking of LayerNorm and residual connections leads to a critical architecture for any initialization.
Bayesian method reconstructs neural network memories from connectivity.
problem Reconstructing memories from neural network connectivity.
method Bayesian inference using statistical physics principles.
result Algorithm successfully reconstructs stored patterns from synaptic connectivity.
Studying neural connectivity is considered one of the most promising and challenging areas of modern neuroscience. The underpinnings of cognition are hidden in the way neurons interact with each other. However, our experimental methods of studying real neural connections at a microscopic level are still arduous and cos…
Deep networks with a wide layer ensure sublevel set connectivity.
problem Ensuring connectivity of sublevel sets in deep learning.
method Analyzing the connectivity of sublevel sets in deep neural networks with a specific layer width.
result A single wide layer of width N+1 suffices to prove connectivity of sublevel sets. Studies in recent years have demonstrated that neural organization and structure impact an individual's ability to perform a given task. Specifically, individuals with greater neural efficiency have been shown to outperform those with less organized functional structure. In this work, we compare the predictive ability …
Neural networks' optimization dynamics are confined to a single basin despite connected basins in the loss landscape.
problem Neural networks' optimization dynamics are confined to a single basin despite connected basins in the loss landscape.
method Identifying entropic barriers arising from the interplay between curvature variations along low-loss paths and noise in optimization dynamics.
result Curvature-induced entropic forces bias noisy dynamics back toward the endpoints, explaining the confinement and connectivity of solutions.
The past few years have witnessed the fast development of different regularization methods for deep learning models such as fully-connected deep neural networks (DNNs) and Convolutional Neural Networks (CNNs). Most of previous methods mainly consider to drop features from input data and hidden layers, such as Dropout, …
Deep neural networks are a powerful tool for feature learning and extraction given their ability to model high-level abstractions in highly complex data. One area worth exploring in feature learning and extraction using deep neural networks is efficient neural connectivity formation for faster feature learning and extr…
Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them. But also generic hardware and software implementations of deep learning run more efficiently for sparse networks. Several methods exist for pruning connections of a neural network after it was trained without conne…
This study connects ReLU neural networks to toric geometry to analyze function realization.
problem Determining which continuous piecewise linear functions can be realized by ReLU neural networks.
method Established a connection between toric geometry and ReLU neural networks, defining key structures like the ReLU fan, toric variety, and Cartier divisor.
result Proved a criterion for functions realizable by unbiased shallow ReLU networks using intersection numbers.
Efficiently estimates neural connectivity using ensemble stimulation.
problem Estimating functional connectivity in large, behaving neural populations.
method Noisy group testing combined with convex optimization and Bayesian inference.
result Connectivity can be inferred with logarithmic growth in tests, even for large networks.
New methods connect low-loss points on neural network surfaces.
problem Connecting low-loss points on neural network loss surfaces.
method Macroscopic distributional assumptions and global connection models.
result Accuracy correlates with complexity and sensitivity.
We draw a formal connection between using synthetic training data to optimize neural network parameters and approximate, Bayesian, model-based reasoning. In particular, training a neural network using synthetic data can be viewed as learning a proposal distribution generator for approximate inference in the synthetic-d…
Residual connections significantly boost the performance of deep neural networks. However, there are few theoretical results that address the influence of residuals on the hypothesis complexity and the generalization ability of deep neural networks. This paper studies the influence of residual connections on the hypoth…
A simple thresholding technique improves graph selection in neural connectivity studies.
problem Graphical model selection for functional neural connectivity in the presence of latent variables.
method Apply a hard thresholding operator to graphical Lasso, neighborhood selection, or CLIME estimators.
result Thresholded estimators outperform existing methods in graph selection consistency and empirical results.
Paper studies ResNet dynamics using NTH, reducing width requirement.
problem Understanding ResNet dynamics and improving training efficiency.
method Uses Neural Tangent Hierarchy (NTH) to analyze ResNet dynamics.
result Reduces width requirement from quartic to cubic for ResNet.
A new method uses neural tangent kernel to efficiently compute MMD statistic.
problem Efficiently computing Maximum Mean Discrepancy (MMD) statistic with low memory and computational complexity.
method Identifies a connection between neural tangent kernel (NTK) and MMD to develop a computationally and memory-efficient approach.
result The proposed NTK-MMD statistic is validated through numerical experiments on synthetic and real-world datasets.
New methods improve neural connectivity analysis at submillisecond timescales.
problem Limitations of standard spike train analysis methods in terms of temporal resolution and scalability.
method Developed Monte Carlo and polynomial approximation methods for continuous-time neural spike train analysis.
result Superior accuracy and scalability compared to traditional binned GLMs, enabling precise connectivity inference.
New algorithms reduce communication in GNN training.
problem Higher communication costs in GNNs due to sparse connectivity.
method Parallel algorithms for sparse-dense matrix multiplication.
result Asymptotic reduction in communication compared to previous methods.
New insights into GNN optimization reveal skip connections and depth accelerate training.
problem Understanding and optimizing the training of Graph Neural Networks (GNNs).
method Analysis of gradient dynamics in linearized GNNs and empirical validation.
result GNNs are implicitly accelerated by skip connections, more depth, and good label distribution during training.
Proposes neuron alignment to optimize mode connectivity in neural networks.
problem Understanding and optimizing mode connectivity in deep neural networks.
method Introduces neuron alignment to approximate optimal weight permutations and improve mode connectivity.
result Neuron alignment significantly alleviates robust loss barriers and improves model robustness and accuracy.
New framework explains deep neural networks using variational spline theory.
problem Understanding functions learned by deep neural networks.
method Developed a variational framework and function space.
result Deep ReLU networks are solutions to regularized data fitting problems over the proposed function space.
A layered neural network is now one of the most common choices for the prediction of high-dimensional practical data sets, where the relationship between input and output data is complex and cannot be represented well by simple conventional models. Its effectiveness is shown in various tasks, however, the lack of inter…
GNNs improve brain activity forecasting in fMRI studies.
problem Understanding neural dynamics in the brain.
method Comparison of GNN architectures for modeling fMRI data.
result GNNs outperform VAR models in robustly scaling to large network studies.
We embed KKT points in neural networks of different sizes.
problem Classifying data using homogeneous neural networks.
method Introducing KKT point embedding principle and proving it for different network types.
result KKT points of a smaller network can be mapped to those of a larger network via linear transformations.