New method finds efficient sparse neural networks.
problem Finding efficient, sparse deep neural network models.
method Continuous Sparsification, approximating regularization.
result Surpasses state-of-the-art for pruning and ticket search.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “continuous sparsification”
Bayesian sparsification improves complex-valued neural networks by 50-100x with minimal performance loss.
problem Efficiently compressing complex-valued neural networks for embedded systems.
method Extending Sparse Variational Dropout to complex-valued networks and conducting a numerical study.
result Achieved state-of-the-art performance on MusicNet with 50-100x compression.
Spectral graph sparsification preserves geometry of GNN embeddings.
problem Maintaining geometric properties of graph neural network embeddings during sparsification.
method Proving spectral sparsification preserves squared pairwise distances, class means, and covariance structure in embedding space.
result Spectral sparsification preserves the geometry of learned embeddings in GNNs.
Gradient Sparsification Can Improve Performance of Differentially-Private Convex Machine Learningcs.LG
Gradient sparsification enhances privacy-preserving machine learning models.
problem Improving performance of differentially-private machine learning models under privacy constraints.
method Gradient sparsification combined with compressed sensing and additive Laplace noise.
result Gradient sparsification can improve performance of differentially-private machine learning models for small privacy budgets.
Sparsification improves convergence in asynchronous distributed SGD, even in the presence of staleness.
problem Staleness in asynchronous distributed SGD.
method Applied sparsification to reduce communication overheads in distributed asynchronous settings.
result The ergodic convergence rate of sparsified asynchronous SGD matches that of vanilla SGD, $\mathcal{O} \left( 1/\sqrt{T}
ight)$, even in the presence of staleness.
FastGAT reduces GNN computation time by 10x using graph sparsification.
problem High computational burden in attention-based GNNs.
method Spectral sparsification to generate optimal graph pruning.
result Per-epoch time is almost linear in graph nodes, reducing computational time by up to 10x.
Empirical error estimates improve graph sparsification reliability.
problem Uncertainty in sparsification error limits downstream computations reliability.
method Data-driven approach to compute empirical error estimates.
result Empirical error estimates provide theoretical guarantees and are computationally feasible.
Many machine learning frameworks, such as resource-allocating networks, kernel-based methods, Gaussian processes, and radial-basis-function networks, require a sparsification scheme in order to address the online learning paradigm. For this purpose, several online sparsification criteria have been proposed to restrict …
Top-k sparsification reduces deep learning communication costs.
problem Reducing communication overhead in distributed deep learning.
method Extensive experiments and theoretical analysis of Top-k sparsification.
result A tighter bound for Top-k operator derived, improving scaling efficiency.
A new federated learning framework with sparsification and adaptive optimization for privacy and efficiency.
problem Lack of sufficient privacy protection in federated learning.
method Integrates random sparsification with gradient perturbation and acceleration techniques to enhance privacy and efficiency.
result Outperforms previous differentially-private federated learning approaches in privacy and efficiency.
We show implicit filter level sparsity manifests in convolutional neural networks (CNNs) which employ Batch Normalization and ReLU activation, and are trained with adaptive gradient descent techniques and L2 regularization or weight decay. Through an extensive empirical study (Mehta et al., 2019) we hypothesize the mec…
A new method to simplify deep neural networks by removing unnecessary parts.
problem Overly complex deep neural networks require significant resource investment for size reduction.
method A fully differentiable sparsification method that optimizes a regularized objective function with stochastic gradient descent.
result The method can learn both the sparsified structure and weights of a network in an end-to-end manner.
Distributed model training suffers from communication overheads due to frequent gradient updates transmitted between compute nodes. To mitigate these overheads, several studies propose the use of sparsified stochastic gradients. We argue that these are facets of a general sparsification method that can operate on any p…
We sparsify gated RNNs by simplifying their structure.
problem Improving efficiency of RNNs by reducing their complexity.
method Adjust existing sparsification techniques to gated RNNs, sparsifying preactivations of gates.
result Simplified LSTM structure improves model performance and efficiency.
Bayesian sparsification reduces deep neural network complexity.
problem Complexity of deep neural networks limits their performance.
method Combines Bayesian shrinkage priors with stochastic variational inference.
result Bayesian model reduction (BMR) is a more efficient alternative for pruning model weights.
A new method converts neural networks to function space for scalable sequential learning.
problem Challenges in gradient-based deep learning for sequential data.
method Dual parameterization of neural networks from weight to function space.
result Efficient scaling, knowledge retention, and new data incorporation.
Modern large scale machine learning applications require stochastic optimization algorithms to be implemented on distributed computational architectures. A key bottleneck is the communication overhead for exchanging information such as stochastic gradients among different workers. In this paper, to reduce the communica…
This paper improves GNN efficiency for large-scale graph applications.
problem High memory usage and computational costs in large-scale graph applications.
method Sparsification techniques from Network Science and Machine Learning.
result Adaptive rewiring enhances GNN performance and scalability.
Spectral sparsification improves Gaussian graphical models under MTP2 constraints.
problem Learning accurate, sparse graphs from data under MTP2 constraints.
method Spectral graph sparsification applied to Gaussian graphical models.
result Spectral-MTP2 preserves MTP2 and approximates the original model well.
New method adapts neural networks without losing prior knowledge.
problem Understanding and enabling flexible adaptation of neural networks.
method Differential geometry framework, functionally invariant paths (FIP).
result Achieves comparable state-of-the-art performance on continual learning and sparsification tasks.
Efficiently sparsifies simplicial complexes using local densities of states.
problem Prohibitive computational requirements for dense simplicial complexes.
method Probabilistic sparsification using local densities of states and kernel-ignoring decomposition.
result Approximates the spectrum of the original SC with a sparser surrogate SC.
New sparsification technique for SGD reduces communication costs.
problem High communication costs in distributed SGD for large-scale models.
method Statistical estimation model for sparsity and skewness of stochastic gradients.
result Concatenated top-k and random-k sparsification outperforms individual methods.
This paper improves the scalability of sparse neural network compression.
problem Sparse neural network compression for diverse data modalities.
method State-of-the-art sparsification techniques and meta-learning.
result Meta-learning sparse compression networks achieve new state-of-the-art results.
New algorithm improves distributed SGD with random sparsification for better convergence and generalization.
problem Communication bottleneck in distributed deep learning.
method Proposes detached error feedback (DEF) algorithm to improve convergence and generalization of communication-efficient distributed SGD.
result Shows better convergence and generalization bounds than existing methods.
In this paper, we investigate effective sketching schemes via sparsification for high dimensional multilinear arrays or tensors. More specifically, we propose a novel tensor sparsification algorithm that retains a subset of the entries of a tensor in a judicious way, and prove that it can attain a given level of approx…
LAGS-SGD optimizes deep learning training by sparsifying gradients layer-wise.
problem Reduces long training times in large deep neural networks with distributed S-SGD.
method Layer-wise adaptive gradient sparsification combined with S-SGD.
result LAGS-SGD achieves convergence guarantees and outperforms vanilla S-SGD.
S-VNNs improve VNNs by sparsifying covariance matrices.
problem Spurious correlations in covariance matrices degrade VNNs' performance and efficiency.
method Apply sparsification techniques on sample covariance matrix and integrate into VNN architecture.
result S-VNNs achieve improved performance, stability, and reduced computational time.
Bayesian approach sparsifies neural networks efficiently.
problem Efficiently pruning neural networks to save resources.
method Sparsifiability via the Marginal likelihood (SpaM) framework.
result Prunes neural networks effectively without significant loss in performance.
New sparsification theorem for Gaussian processes reduces dimensionality.
problem Dimension-independent sparsification of Gaussian process suprema.
method Dimension-independent sparsification of Gaussian process suprema.
result Sparsifier size is independent of and .
New architectures improve KANs, making them more interpretable and accurate.
problem Improving Kolmogorov-Arnold networks while maintaining interpretability.
method Overprovisioned architectures combined with sparsification, deep supervision, and depth selection, optimized with a minimum description length objective.
result Combining sparsification with depth selection achieves competitive or superior accuracy while discovering smaller models.
Jointly learns feature and sample relevancies for robust sparse recovery.
problem Sparse recovery sensitivity to data contaminants like outliers or misspecified noise.
method Jointly learns feature and sample relevancies via marginal likelihood optimization.
result Consistent sparse and robust prediction models across diverse tasks.
In this paper, we provide a novel construction of the linear-sized spectral sparsifiers of Batson, Spielman and Srivastava [BSS14]. While previous constructions required running time [BSS14, Zou12], our sparsification routine can be implemented in almost-quadratic running time . The funda…
The Minimum Description Length (MDL) principle states that the optimal model for a given data set is that which compresses it best. Due to practial limitations the model can be restricted to a class such as linear regression models, which we address in this study. As in other formulations such as the LASSO and forward …
While the harmonic function solution performs well in many semi-supervised learning (SSL) tasks, it is known to scale poorly with the number of samples. Recent successful and scalable methods, such as the eigenfunction method focus on efficiently approximating the whole spectrum of the graph Laplacian constructed from …
Huge scale machine learning problems are nowadays tackled by distributed optimization algorithms, i.e. algorithms that leverage the compute power of many devices for training. The communication overhead is a key bottleneck that hinders perfect scalability. Various recent works proposed to use quantization or sparsifica…
This paper is concerned with the sparsification of the input-hidden weights of ELM (Extreme Learning Machine). For ordinary feedforward neural networks, the sparsification is usually done by introducing certain regularization technique into the learning process of the network. But this strategy can not be applied for E…
Bayesian methods have been successfully applied to sparsify weights of neural networks and to remove structure units from the networks, e. g. neurons. We apply and further develop this approach for gated recurrent architectures. Specifically, in addition to sparsification of individual weights and neurons, we propose t…
We introduce Continual Learning via Neural Pruning (CLNP), a new method aimed at lifelong learning in fixed capacity models based on neuronal model sparsification. In this method, subsequent tasks are trained using the inactive neurons and filters of the sparsified network and cause zero deterioration to the performanc…
We analyze the size of the dictionary constructed from online kernel sparsification, using a novel formula that expresses the expected determinant of the kernel Gram matrix in terms of the eigenvalues of the covariance operator. Using this formula, we are able to connect the cardinality of the dictionary with the eigen…
Distributed training of massive machine learning models, in particular deep neural networks, via Stochastic Gradient Descent (SGD) is becoming commonplace. Several families of communication-reduction methods, such as quantization, large-batch methods, and gradient sparsification, have been proposed. To date, gradient s…
One-bit quantization and sparsification improve multiclass classification with strong regularization.
problem Overfitting mislabeled data in multiclass classification.
method Linear regression with regularization and one-bit quantization/sparsification.
result Sparse and one-bit solutions perform almost as well as the optimal solution with .
Estimates mean of distributed vectors with sparsification and spatial/temporal correlations.
problem Estimating mean of high-dimensional vectors distributed across nodes with low communication cost.
method Modifies decoding method to leverage spatial and temporal correlations in sparsified vectors.
result Estimators consistently outperform more sophisticated sparsification methods.
Wasserstein Measure Coresetsstat.ML
The proliferation of large data sets and Bayesian inference techniques motivates demand for better data sparsification. Coresets provide a principled way of summarizing a large dataset via a smaller one that is guaranteed to match the performance of the full data set on specific problems. Classical coresets, however, n…
In natural language processing, a lot of the tasks are successfully solved with recurrent neural networks, but such models have a huge number of parameters. The majority of these parameters are often concentrated in the embedding layer, which size grows proportionally to the vocabulary length. We propose a Bayesian spa…
New method grows deep networks efficiently by dynamically pruning and growing layers.
problem Training deep networks is computationally expensive and inefficient.
method Structured continuous sparsification starting from a small seed architecture.
result 49.7% inference FLOPs and 47.4% training FLOPs savings with 75.2% top-1 accuracy.
Accelerates GPR with localized kernels for faster performance.
problem Speeding up Gaussian process regression.
method Localization kernels applied at each data point to down-weight distant points, leading to a sparsified Gram matrix.
result Significant speedups with competitive performance compared to other methods.
Spectral sparsification improves Laplacian-constrained graph learning.
problem Improving accuracy of Laplacian-constrained graph learning.
method Spectral graph sparsification as a post-estimation operation.
result Improved accuracy of Laplacian-constrained graph learning.
A new method efficiently approximates Gromov-Wasserstein distance.
problem High computational complexity of Gromov-Wasserstein distance.
method Importance sparsification method to construct a sparse coupling matrix.
result Efficient approximation of GW distance with reduced complexity.