SSVI efficiently trains sparse Bayesian neural networks with minimal compression and performance loss.
problem Efficiently training Bayesian neural networks with uncertainty quantification.
method SSVI optimizes a sparse subspace basis selection and its parameters alternately, guided by weight distribution statistics.
result SSVI achieves significant compression (10-20x model size reduction) with minimal performance drop (under 3%) and FLOPs reduction (up to 20x) compared to dense Variational Inference.
Campfire simplifies sparse training for CNNs, achieving high accuracy with minimal pruning.
problem Training CNNs with sparse weights to maintain high accuracy and robustness.
method Gradual pruning technique with simplified sparsity structure, exploring various tradeoffs.
result Sparse ResNet-50 and ResNet-50 v1.5 achieve 75% top-1 accuracy with 70% sparsity, <1% accuracy loss.
The goal of predictive sparse coding is to learn a representation of examples as sparse linear combinations of elements from a dictionary, such that a learned hypothesis linear in the new representation performs well on a predictive task. Predictive sparse coding algorithms recently have demonstrated impressive perform…
NeuroFabric proposes a method to optimize sparse network training topologies.
problem Long training times in deep neural networks due to high memory and compute requirements.
method Developed a new sparse neural network initialization scheme and evaluated various topologies.
result Identified a single optimal topology that maximizes accuracy across different datasets.
Sparse coding has been popularly used as an effective data representation method in various applications, such as computer vision, medical imaging and bioinformatics, etc. However, the conventional sparse coding algorithms and its manifold regularized variants (graph sparse coding and Laplacian sparse coding), learn th…
Sparse models help in selecting fewer variables for efficient predictions.
problem Overfitting and high computational costs in learning models.
method Automated variable selection for sparse predictive models.
result Sparse models improve model efficiency and interpretability.
Sparse routing networks with co-training prevent catastrophic forgetting in continual learning.
problem Catastrophic forgetting in neural networks trained on a sequence of tasks.
method Sparse routing networks with co-training to minimize interference between dissimilar tasks.
result Sparse routing networks with co-training outperform densely connected networks on benchmarks.
Novel active learning framework using sparse approximation for efficient model training.
problem Efficient model training with limited labeled data.
method Formulates batch active learning as sparsity-constrained discontinuous optimization problems, using greedy or proximal iterative hard thresholding algorithms.
result Achieves competitive performance with lower computational complexity across different settings.
FedDST trains sparse sub-networks to improve efficiency in federated learning.
problem Efficiently train large models on resource-limited edge devices with limited network bandwidth.
method Dynamic sparse training to reduce model size and communication.
result FedDST outperforms existing methods in federated learning, especially in non-i.i.d. settings.
A new approach to MI learning using bag-to-class divergence.
problem Sparse MI training sets and difficulty in classifying bags.
method Introducing bag-to-class divergence to MI learning, emphasizing hierarchical random vectors.
result Bag-to-class divergence is a more effective classifier for MI learning.
This paper studies CSDL's reconstruction risk and finds it consistent in the ultra-sparse setting.
problem The statistical properties of convolutional sparse dictionary learning (CSDL).
method Identifies the minimax convergence rate of CSDL in terms of reconstruction risk, upper bounds the risk of an established CSDL estimator, and proves a matching lower bound.
result Consistency in reconstruction risk is possible precisely in the ultra-sparse setting.
Sparse coding approximates the data sample as a sparse linear combination of some basic codewords and uses the sparse codes as new presentations. In this paper, we investigate learning discriminative sparse codes by sparse coding in a semi-supervised manner, where only a few training samples are labeled. By using the m…
Dynamic Sparse Training finds efficient sparse networks from scratch.
problem Finding efficient sparse neural networks.
method Jointly optimizes network parameters and sparsity with trainable thresholds.
result Achieves state-of-the-art performance with minimal performance loss.
HERA improves PLL by integrating heterogeneous loss and sparse-low-rank regularization.
problem Learning from data with partial labels.
method Combines heterogeneous loss and sparse-low-rank regularization.
result Achieves superior performance on artificial and real-world data.
Powerpropagation makes neural networks inherently sparse.
problem Training sparse neural networks to reduce computational footprint and model size.
method Introduces a new weight-parameterisation technique exploiting gradient descent dynamics.
result Models trained with Powerpropagation have a higher density of zero weights, allowing for more efficient pruning.
Efficiently trains HDP topic models on large datasets using a sparse data-parallel sampler.
problem Scaling non-parametric topic models to large datasets.
method Data-parallel training with a doubly sparse sampler for HDP topic models.
result Trains HDP topic models on a 8m document, 768m token PubMed corpus in under 4 days.
The paper explains practical insights for sparse network modeling.
problem Resolving pathologies in traditional network modeling, focusing on sparsity.
method Sparse exchangeable graphs, network subsampling, test-train dataset splitting, mean field variational inference.
result Practical insights and methods for sparse network modeling.
Paper develops efficient variational inference for sparse deep learning with theoretical guarantees.
problem Sparse deep learning's challenge of huge storage consumption and sparse structure recovery.
method Bayesian treatment with spike-and-slab priors and continuous relaxation of Bernoulli distribution for computationally efficient variational inferences.
result Provides variational posterior contraction rate, justifying consistency of the proposed method.
Improves inference from sparse data with hybrid summary statistics.
problem Robust simulation-based inference from limited data.
method Augment traditional summary statistics with neural network outputs to maximize mutual information.
result Improves information extraction and makes inference robust in low-data settings.
New method finds efficient sparse neural networks.
problem Finding efficient, sparse deep neural network models.
method Continuous Sparsification, approximating ℓ0 regularization. result Surpasses state-of-the-art for pruning and ticket search.
Sparse learning speeds up neural network training without sacrificing accuracy.
problem Training deep neural networks efficiently while maintaining performance.
method Sparse momentum algorithm that redistributes and grows weights based on momentum magnitude.
result State-of-the-art sparse performance on various datasets with up to 5.61x faster training.
DistGP models multi-robot mapping with distributed Gaussian process learning.
problem Collaborative mapping by multiple robots with limited local data.
method Sparse Gaussian process with factorisation and distributed training via GBP.
result DistGP achieves superior accuracy and robustness compared to DiNNO.
Transformers excel at sparse token selection, surpassing FCNs in both worst and average cases.
problem Sparse token selection task
method One-layer transformer trained with gradient descent
result Transformers learn sparse token selection and exhibit strong out-of-distribution length generalization
ADAPT method controls balance between sparse generator and discriminator in GANs.
problem High computational complexity in GANs training.
method Balanced Dynamic Sparse Training (ADAPT) method.
result ADAPT method achieves a good trade-off between performance and computational cost.
Simple linear model outperforms deep models on sparse data.
problem Sparse data in recommender systems.
method Linear autoencoder with closed-form solution.
result Simple model achieves better ranking accuracy than deep models.
New algorithm generates diverse sparse neural network topologies.
problem Training deep neural networks requires significant hardware resources.
method Deterministic algorithm to generate sparse neural network topologies.
result Generated topologies train to the same precision as dense DNNs at lower cost.
Method generates dense fields from sparse measurements without needing spatial statistics or examples.
problem Generating dense physical fields from sparse measurements.
method Introduces a differentiable numerical simulator into neural network training.
result Superior results on fluid mechanics problems compared to statistical and neural network methods.
Method trains sparse neural networks without sacrificing accuracy.
problem Training sparse neural networks limits model size.
method Updates sparse network topology during training.
result Requires fewer FLOPs to achieve accuracy.
Sparse modeling improves portfolio optimization by reducing errors in complex market systems.
problem Errors in multivariate modeling of markets and economy.
method L0-norm sparse elliptical modeling to reduce oversimplification, and study likelihood in- and out-of-sample for different parameter lengths.
result Sparse models lead to better portfolio performance, higher out-of-sample likelihood, and lower volatility.
New method speeds up sparse graph neural networks training on dense hardware.
problem Training sparse graph neural networks is slow on custom hardware.
method Inspired by sparse matrix optimization, developed techniques for dense hardware.
result Sparse graph neural networks trained in 13 minutes on 512-core TPUv2 Pod.
Study shows how specialized attention circuits emerge during transformer training.
problem Understanding the mechanisms of transformer training dynamics at large scales.
method Controlled sparse modular addition task; monitoring token evolution via visual sandbox.
result Specialized attention circuits (clustering heads) naturally emerge during training.
Sparse neural networks training is difficult due to optimization failures and energy landscape issues.
problem Training sparse neural networks leads to suboptimal solutions and optimization failures.
method Investigated optimization dynamics and energy landscape in sparse neural networks.
result Sparse neural networks have a linear path with a monotonically decreasing objective from initialization to a good solution, but not from a bad solution.
Framework models graph-connected entities with sparse shared HMMs.
problem Model sequential data from graph-connected entities.
method Sparse Mixture of Hidden Markov Models (HMMs) trained jointly with graph topology.
result Effectiveness and versatility demonstrated in experiments.
RandNet learns from compressed image data, improving efficiency and accuracy.
problem Efficiency and accuracy in training neural networks with large datasets.
method RandNet uses compressed random measurements of images to train neural networks efficiently.
result RandNet achieves comparable accuracy to full data training with minimal loss.
SGD learns sparse parities near computational limits with discontinuous phase transitions.
problem Learning sparse parities in deep learning.
method Empirical and theoretical analysis of SGD on sparse parities.
result SGD makes progress on sparse parities via Fourier gap, not Langevin-like mechanism.
VolNP learns IVS from sparse quotes via meta-learning and SABR priors.
problem Reconstructing implied volatility surfaces from sparse option quotes.
method Meta-learning Neural Process with SABR-induced priors.
result VolNP outperforms SABR, SSVI, and Gaussian process on SPX options.
New online method learns from all images, supports incomplete data.
problem High memory usage and limited training data in batch methods.
method Online convolutional dictionary learning with spatial mask support.
result Improved performance and better scalability with training set size.
RadiX-Net generates diverse sparse neural topologies.
problem Sparse neural networks require more efficient storage and training.
method Deterministically generates RadiX-Nets from sparse topologies.
result RadiX-Nets can train to the same precision as dense DNNs at lower cost.
Quantum SVM uses fewer features for faster training.
problem Training high-dimensional SVMs efficiently.
method Quantum linear programming for sparse SVM training.
result Quantum sparse SVM can be trained in sublinear time.
Sparse nearest-centroid classifiers detect relevant features for classification.
problem Classifying data with low computational cost and feature selection.
method Proposes ℓ1 and ℓ2 sparse variants of nearest-centroid classifiers. result Training sparse classifiers can be done exactly and at quasi-linear cost.
Sparse Transformers reduce memory and time requirements for long sequence modeling.
problem Quadratic growth in memory and time with transformer sequence length.
method Sparse factorizations of attention matrix, deeper network training, attention matrix recomputation, fast attention kernels.
result Sparse Transformers can model sequences up to tens of thousands of timesteps with hundreds of layers.
SWAT reduces training efficiency and speed for CNNs.
problem Efficient training of neural networks on sparse hardware.
method SWAT modifies backpropagation to eliminate small weights and activations.
result SWAT reduces FLOPS by 80% on ResNet-50 for ImageNet, improving training speed by 3.3x.
Regularization improves stability and consistency of sparse autoencoders.
problem Varying features across random seeds and training choices in SAEs.
method Added L1 or L2 penalties on encoder and decoder weights.
result L2 regularization increases cross-seed feature consistency.
Sparse sub-networks win transfer learning tasks.
problem Transfer learning with deep networks.
method Unstructured magnitude pruning to find winning tickets.
result Sparse sub-networks achieve similar or better accuracy than original networks.
Sparse neural networks can improve performance with less memory.
problem Lack of fast memory limits deep neural network performance.
method Experimented with sparse neural network topologies, including pruning-based and RadiX-Nets.
result Sparse networks achieve comparable accuracy to dense networks but suffer instability at extreme sparsity.
Deep Rewiring trains very sparse neural networks efficiently.
problem Limits of connectivity in neuromorphic hardware.
method DEEP R algorithm that trains sparsely connected neural networks directly.
result Trains very sparse feedforward and recurrent neural networks with minimal performance loss.
New pruning methods improve dynamic sparse training performance.
problem Improving dynamic sparse training performance.
method Design and empirical analysis of pruning criteria.
result Most pruning methods yield similar results, but magnitude-based pruning performs best in low-density regimes.
Sparse support vector machine (SVM) is a popular classification technique that can simultaneously learn a small set of the most interpretable features and identify the support vectors. It has achieved great successes in many real-world applications. However, for large-scale problems involving a huge number of samples a…