Spectral graph sparsification preserves geometry of GNN embeddings.
problem Maintaining geometric properties of graph neural network embeddings during sparsification.
method Proving spectral sparsification preserves squared pairwise distances, class means, and covariance structure in embedding space.
result Spectral sparsification preserves the geometry of learned embeddings in GNNs.
A new method to simplify deep neural networks by removing unnecessary parts.
problem Overly complex deep neural networks require significant resource investment for size reduction.
method A fully differentiable sparsification method that optimizes a regularized objective function with stochastic gradient descent.
result The method can learn both the sparsified structure and weights of a network in an end-to-end manner.
FastGAT reduces GNN computation time by 10x using graph sparsification.
problem High computational burden in attention-based GNNs.
method Spectral sparsification to generate optimal graph pruning.
result Per-epoch time is almost linear in graph nodes, reducing computational time by up to 10x.
We sparsify gated RNNs by simplifying their structure.
problem Improving efficiency of RNNs by reducing their complexity.
method Adjust existing sparsification techniques to gated RNNs, sparsifying preactivations of gates.
result Simplified LSTM structure improves model performance and efficiency.
Many machine learning frameworks, such as resource-allocating networks, kernel-based methods, Gaussian processes, and radial-basis-function networks, require a sparsification scheme in order to address the online learning paradigm. For this purpose, several online sparsification criteria have been proposed to restrict …
This paper improves GNN efficiency for large-scale graph applications.
problem High memory usage and computational costs in large-scale graph applications.
method Sparsification techniques from Network Science and Machine Learning.
result Adaptive rewiring enhances GNN performance and scalability.
We show implicit filter level sparsity manifests in convolutional neural networks (CNNs) which employ Batch Normalization and ReLU activation, and are trained with adaptive gradient descent techniques and L2 regularization or weight decay. Through an extensive empirical study (Mehta et al., 2019) we hypothesize the mec…
Bayesian sparsification reduces deep neural network complexity.
problem Complexity of deep neural networks limits their performance.
method Combines Bayesian shrinkage priors with stochastic variational inference.
result Bayesian model reduction (BMR) is a more efficient alternative for pruning model weights.
Bayesian sparsification improves complex-valued neural networks by 50-100x with minimal performance loss.
problem Efficiently compressing complex-valued neural networks for embedded systems.
method Extending Sparse Variational Dropout to complex-valued networks and conducting a numerical study.
result Achieved state-of-the-art performance on MusicNet with 50-100x compression.
Bayesian approach sparsifies neural networks efficiently.
problem Efficiently pruning neural networks to save resources.
method Sparsifiability via the Marginal likelihood (SpaM) framework.
result Prunes neural networks effectively without significant loss in performance.
New method finds efficient sparse neural networks.
problem Finding efficient, sparse deep neural network models.
method Continuous Sparsification, approximating ℓ0 regularization. result Surpasses state-of-the-art for pruning and ticket search.
S-VNNs improve VNNs by sparsifying covariance matrices.
problem Spurious correlations in covariance matrices degrade VNNs' performance and efficiency.
method Apply sparsification techniques on sample covariance matrix and integrate into VNN architecture.
result S-VNNs achieve improved performance, stability, and reduced computational time.
Distributed model training suffers from communication overheads due to frequent gradient updates transmitted between compute nodes. To mitigate these overheads, several studies propose the use of sparsified stochastic gradients. We argue that these are facets of a general sparsification method that can operate on any p…
A new federated learning framework with sparsification and adaptive optimization for privacy and efficiency.
problem Lack of sufficient privacy protection in federated learning.
method Integrates random sparsification with gradient perturbation and acceleration techniques to enhance privacy and efficiency.
result Outperforms previous differentially-private federated learning approaches in privacy and efficiency.
Modern large scale machine learning applications require stochastic optimization algorithms to be implemented on distributed computational architectures. A key bottleneck is the communication overhead for exchanging information such as stochastic gradients among different workers. In this paper, to reduce the communica…
New architectures improve KANs, making them more interpretable and accurate.
problem Improving Kolmogorov-Arnold networks while maintaining interpretability.
method Overprovisioned architectures combined with sparsification, deep supervision, and depth selection, optimized with a minimum description length objective.
result Combining sparsification with depth selection achieves competitive or superior accuracy while discovering smaller models.
Bayesian methods have been successfully applied to sparsify weights of neural networks and to remove structure units from the networks, e. g. neurons. We apply and further develop this approach for gated recurrent architectures. Specifically, in addition to sparsification of individual weights and neurons, we propose t…
Gradient sparsification enhances privacy-preserving machine learning models.
problem Improving performance of differentially-private machine learning models under privacy constraints.
method Gradient sparsification combined with compressed sensing and additive Laplace noise.
result Gradient sparsification can improve performance of differentially-private machine learning models for small privacy budgets.
Spectral sparsification improves Gaussian graphical models under MTP2 constraints.
problem Learning accurate, sparse graphs from data under MTP2 constraints.
method Spectral graph sparsification applied to Gaussian graphical models.
result Spectral-MTP2 preserves MTP2 and approximates the original model well.
Sparsification improves convergence in asynchronous distributed SGD, even in the presence of staleness.
problem Staleness in asynchronous distributed SGD.
method Applied sparsification to reduce communication overheads in distributed asynchronous settings.
result The ergodic convergence rate of sparsified asynchronous SGD matches that of vanilla SGD, $\mathcal{O} \left( 1/\sqrt{T}
ight)$, even in the presence of staleness.
This paper is concerned with the sparsification of the input-hidden weights of ELM (Extreme Learning Machine). For ordinary feedforward neural networks, the sparsification is usually done by introducing certain regularization technique into the learning process of the network. But this strategy can not be applied for E…
Empirical error estimates improve graph sparsification reliability.
problem Uncertainty in sparsification error limits downstream computations reliability.
method Data-driven approach to compute empirical error estimates.
result Empirical error estimates provide theoretical guarantees and are computationally feasible.
The paper sparsifies networks by finding efficient paths in their functional space.
problem Sparsifying neural networks to improve performance and efficiency.
method The authors use the geometry of weight spaces and functional manifolds to find efficient paths (geodesics) in the functional space of neural networks.
result The proposed framework can sparsify networks and improve performance on various tasks.
Top-k sparsification reduces deep learning communication costs.
problem Reducing communication overhead in distributed deep learning.
method Extensive experiments and theoretical analysis of Top-k sparsification.
result A tighter bound for Top-k operator derived, improving scaling efficiency.
LAGS-SGD optimizes deep learning training by sparsifying gradients layer-wise.
problem Reduces long training times in large deep neural networks with distributed S-SGD.
method Layer-wise adaptive gradient sparsification combined with S-SGD.
result LAGS-SGD achieves convergence guarantees and outperforms vanilla S-SGD.
While variational dropout approaches have been shown to be effective for network sparsification, they are still suboptimal in the sense that they set the dropout rate for each neuron without consideration of the input data. With such input-independent dropout, each neuron is evolved to be generic across inputs, which m…
Adaptive dropout and regularization are shown to be dual in linear networks.
problem Sparsifying deep neural networks.
method Examining dropout in the linear case, revealing a duality with regularization.
result Adaptive dropout methods lead to sparse solutions with effective penalties similar to classical sparse optimization penalties.
We develop a sparse representation method for neural network uncertainty.
problem Estimating model uncertainty in neural networks.
method Sparse representation of model uncertainty using inverse Multivariate Normal Distribution (MND), with a novel sparsification algorithm and analytical sampler.
result The information form of neural networks can be effectively applied for model uncertainty representation, showing competitive performance.
In natural language processing, a lot of the tasks are successfully solved with recurrent neural networks, but such models have a huge number of parameters. The majority of these parameters are often concentrated in the embedding layer, which size grows proportionally to the vocabulary length. We propose a Bayesian spa…
Model neural plasticity as binary optimization to dynamically activate or deactivate network units.
problem Dynamic learning and adaptability of neural networks.
method Model neural plasticity as an L0-norm regularized binary optimization problem, where units can be activated or deactivated based on a cost-benefit tradeoff. result Demonstrates that a single parameter k can modulate learning dynamics, unifying network sparsification and expansion. Proposes a GNN for multivariate time-series prediction with filtering.
problem Low signal-to-noise ratio in complex systems data.
method Integrates a spatial-temporal GNN with a matrix filtering module to generate filtered graphs.
result Proposed model outperforms baseline approaches in multivariate time-series prediction.
S2KAN integrates symbolic primitives into neural network activations for improved interpretability.
problem Training activations in KANs often lack symbolic fidelity, leading to unintelligible models.
method Softly Symbolified Kolmogorov-Arnold Networks (S2KAN) integrates symbolic primitives into training with learnable gates and a Minimum Description Length objective.
result S2KAN discovers interpretable forms when symbolic terms suffice, gracefully degrading to dense splines when necessary.
A new method converts neural networks to function space for scalable sequential learning.
problem Challenges in gradient-based deep learning for sequential data.
method Dual parameterization of neural networks from weight to function space.
result Efficient scaling, knowledge retention, and new data incorporation.
Distributed training of massive machine learning models, in particular deep neural networks, via Stochastic Gradient Descent (SGD) is becoming commonplace. Several families of communication-reduction methods, such as quantization, large-batch methods, and gradient sparsification, have been proposed. To date, gradient s…
Paper tackles regularization and sparsification for quaternion neural networks.
problem Regularizing and sparsifying quaternion neural networks for compactness and real-time applications.
method Developed targeted regularization strategies for quaternion neural networks, extending l1 and structured regularization.
result Tailored strategies significantly reduce the number of connections and neurons, resulting in smaller, more compact networks.
We study epidemic forecasting on real-world health data by a graph-structured recurrent neural network (GSRNN). We achieve state-of-the-art forecasting accuracy on the benchmark CDC dataset. To improve model efficiency, we sparsify the network weights via transformed-ℓ1 penalty and maintain prediction accuracy at…
Hierarchical FL reduces latency in HCNs by sharing model updates.
problem Latency and privacy issues in federated learning across heterogeneous cellular networks.
method Hierarchical federated learning, gradient sparsification, periodic averaging.
result Significant reduction in communication latency without compromising model accuracy.
Graphs from features improve classification accuracy in tasks.
problem Traditional classification tasks can be improved by incorporating relational information.
method Construct geometric graphs from features and use them in Graph Convolutional Networks.
result Graphs derived from features increase classification accuracy and improve class separation.
New algorithm reduces communication traffic in decentralized learning.
problem Communication bottleneck in decentralized learning for low-bandwidth workers.
method Sparsification and adaptive peer selection to reduce communication traffic.
result Significant reduction in communication traffic compared to existing methods.
This paper improves the scalability of sparse neural network compression.
problem Sparse neural network compression for diverse data modalities.
method State-of-the-art sparsification techniques and meta-learning.
result Meta-learning sparse compression networks achieve new state-of-the-art results.
This paper proves a generalization bound for complex-valued neural networks scaling with spectral complexity.
problem Ensuring the performance of complex-valued neural networks on unseen data.
method Theoretical derivation using Maurey Sparsification Lemma and Dudley Entropy Integral, empirical validation on various datasets.
result The spectral complexity of weight matrices is a significant factor in the generalization ability of complex-valued neural networks.
We rigorously evaluate three state-of-the-art techniques for inducing sparsity in deep neural networks on two large-scale learning tasks: Transformer trained on WMT 2014 English-to-German, and ResNet-50 trained on ImageNet. Across thousands of experiments, we demonstrate that complex techniques (Molchanov et al., 2017;…
Efficiently sparsifies simplicial complexes using local densities of states.
problem Prohibitive computational requirements for dense simplicial complexes.
method Probabilistic sparsification using local densities of states and kernel-ignoring decomposition.
result Approximates the spectrum of the original SC with a sparser surrogate SC.
New sparsification technique for SGD reduces communication costs.
problem High communication costs in distributed SGD for large-scale models.
method Statistical estimation model for sparsity and skewness of stochastic gradients.
result Concatenated top-k and random-k sparsification outperforms individual methods.
Spectral sparsification improves Laplacian-constrained graph learning.
problem Improving accuracy of Laplacian-constrained graph learning.
method Spectral graph sparsification as a post-estimation operation.
result Improved accuracy of Laplacian-constrained graph learning.
New algorithm improves distributed SGD with random sparsification for better convergence and generalization.
problem Communication bottleneck in distributed deep learning.
method Proposes detached error feedback (DEF) algorithm to improve convergence and generalization of communication-efficient distributed SGD.
result Shows better convergence and generalization bounds than existing methods.
In this paper, we investigate effective sketching schemes via sparsification for high dimensional multilinear arrays or tensors. More specifically, we propose a novel tensor sparsification algorithm that retains a subset of the entries of a tensor in a judicious way, and prove that it can attain a given level of approx…
New method preserves spectral clustering performance under aggressive sparsification and quantization.
problem Maintaining spectral clustering performance with sparse and quantized data.
method Random matrix theory applied to eigenspectrum changes under sparsification and quantization.
result Spectral clustering performance is preserved even with aggressive sparsification and quantization.