Sparse Hopfield model improves memory retrieval with fewer connections.
problem Memory retrieval efficiency with fewer connections.
method Sparse extension of Hopfield model, derived from sparse entropic regularizer.
result Sparse Hopfield model achieves tighter error bounds and better performance.
Using a Bayesian approach, we consider the problem of recovering sparse signals under additive sparse and dense noise. Typically, sparse noise models outliers, impulse bursts or data loss. To handle sparse noise, existing methods simultaneously estimate the sparse signal of interest and the sparse noise of no interest.…
Hierarchical Block Sparse Neural Networks improve both accuracy and runtime efficiency of sparse DNNs.
problem Inefficiency of sparse DNNs on regular parallel hardware due to irregular computation.
method Introducing HBsNN, a structured sparse neural network that balances accuracy and runtime efficiency.
result HBsNN achieves better runtime performance and accuracy than unstructured and highly structured sparse models.
Sparse-RS framework efficiently attacks models with sparse perturbations.
problem Efficiently attacking models with sparse perturbations in the black-box setting.
method Random search-based Sparse-RS framework for score-based attacks.
result Sparse-RS achieves state-of-the-art success rate and query efficiency.
New techniques enforce sparseness in recurrent models, reducing memory usage.
problem Reducing memory usage in recurrent sequence models for NLP.
method Enforcing sparseness upfront in recurrent layers for language modeling and sequence labeling.
result Predefined sparseness leads to similar performance with fewer parameters.
New measure SEV shows non-sparse models can still have low decision sparsity.
problem Non-sparse models can still make accurate decisions based on a few features.
method Introduced Sparse Explanation Value (SEV) to measure decision sparsity, not overall model sparsity.
result Many non-sparse models have low decision sparsity, as measured by SEV.
SRF learns sparse rule models by screening out features efficiently.
problem Learning optimal sparse rule models is computationally intractable due to the large number of possible rules.
method SRF uses meta safe screening (mSS) to efficiently screen out multiple features, improving the learning of sparse rule models.
result SRF provides a general framework for fitting sparse rule models and can handle group regularization.
Proposes FARM model combining latent factor and sparse regression.
problem Testing adequacy of latent factor and sparse regression models.
method Factor Augmented sparse linear Regression Model (FARM) with FabTest and ANOVA type tests.
result Model robustness and effectiveness validated through experiments.
Sparse models help in selecting fewer variables for efficient predictions.
problem Overfitting and high computational costs in learning models.
method Automated variable selection for sparse predictive models.
result Sparse models improve model efficiency and interpretability.
Dynamic Sparse Training finds efficient sparse networks from scratch.
problem Finding efficient sparse neural networks.
method Jointly optimizes network parameters and sparsity with trainable thresholds.
result Achieves state-of-the-art performance with minimal performance loss.
Bayesian Lasso Sparse model provides sparse estimates in linear and nonlinear regression.
problem Sparse learning in regression models.
method Develops a new sparse learning model using type-II maximum likelihood procedure.
result The BLS model provides sparse estimates and is more precise, especially with noisy data.
We introduce a new algorithm, called adaptive sparse backfitting algorithm, for solving high dimensional Sparse Additive Model (SpAM) utilizing symmetric, non-negative definite smoothers. Unlike the previous sparse backfitting algorithm, our method is essentially a block coordinate descent algorithm that guarantees to …
This work introduces a method to compare sparse neural network topologies using graph theory.
problem Comparing and understanding sparse neural network topologies, especially during training.
method Introducing Neural Network Sparse Topology Distance (NNSTD) to measure distances between different sparse neural networks.
result Sparse neural networks can outperform over-parameterized models without further structure optimization.
Solves community detection in sparse hypergraphs above a threshold.
problem Community detection in sparse hypergraphs.
method Generalization of Massoulié's method for sparse random graphs to random hypergraphs.
result Above the threshold, a spectral algorithm constructs a partition correlated with the true partition.
This paper explores efficient neural networks by identifying sparse structures.
problem Reducing computational complexity in neural networks without sacrificing accuracy.
method Identifying and utilizing sparse structures in neural network weights and activations.
result Large and sparse models are more beneficial for practical problems.
Evidential Softmax preserves multimodality in sparse probability distributions for generative models.
problem Sparse probability distributions in deep generative models make exact marginalization computationally intractable.
method Introduce ev-softmax, a sparse normalization function that preserves multimodality and can be trained with probabilistic loss functions.
result ev-softmax outperforms existing techniques in distributional accuracy and dimensionality reduction.
Sparse VAE learns latent factors from high-dimensional data.
problem Unsupervised representation learning on high-dimensional data.
method Sparse VAE model that learns latent factors summarizing data associations.
result Sparse VAE can recover true model parameters with infinite data.
Sparse linear models improve neural network debuggability.
problem Improving neural network interpretability and debugging.
method Using sparse linear models over learned deep feature representations.
result The approach leads to more debuggable and accurate neural networks.
We present a comprehensive framework for structured sparse coding and modeling extending the recent ideas of using learnable fast regressors to approximate exact sparse codes. For this purpose, we develop a novel block-coordinate proximal splitting method for the iterative solution of hierarchical sparse coding problem…
Is it possible to find the sparsest vector (direction) in a generic subspace S⊆Rp with dim(S)=n<p? This problem can be considered a homogeneous variant of the sparse recovery problem, and finds connections to sparse dictionary learning, sparse PCA, and many other …
Spectral clustering with edge counting detects communities in sparse models.
problem Detecting communities in sparse latent space models.
method Spectral clustering followed by edge counting.
result Algorithm achieves consistency and optimality for a broad class of models.
Fitting high-dimensional data involves a delicate tradeoff between faithful representation and the use of sparse models. Too often, sparsity assumptions on the fitted model are too restrictive to provide a faithful representation of the observed data. In this paper, we present a novel framework incorporating sparsity i…
Sparse manifold transform linearizes non-linear signal transformations.
problem Non-linear signal transformations in sensory data.
method Combines sparse coding, manifold learning, and slow feature analysis.
result Models sparse discreteness and low-dimensional manifold structure in natural scenes.
DS-Softmax speeds up softmax inference by learning sparse experts.
problem Expensive softmax computations for large output classes.
method Sparse mixture of sparse experts for efficient top-k class retrieval.
result Significant computation reductions achieved at no performance loss.
The paper explains practical insights for sparse network modeling.
problem Resolving pathologies in traditional network modeling, focusing on sparsity.
method Sparse exchangeable graphs, network subsampling, test-train dataset splitting, mean field variational inference.
result Practical insights and methods for sparse network modeling.
Sparse representations using data dictionaries provide an efficient model particularly for signals that do not enjoy alternate analytic sparsifying transformations. However, solving inverse problems with sparsifying dictionaries can be computationally expensive, especially when the dictionary under consideration has a …
Sparse APCA identifies sparse factors in financial returns over time.
problem Analyzing co-movements of high-dimensional panel data over time.
method Sparse asymptotic PCA with truncated power method for sparse factors and sequential deflation for multi-factor cases.
result Identification of nine risk factors influencing the S&P 500 stock market.
This paper proposes a method to reveal task relationships in multi-task learning models using sparse graphs.
problem Understanding the underlying task relationships in multi-task learning models.
method Proposes a bilevel formulation of multi-task learning that induces sparse graphs.
result The method improves interpretability of multi-task learning models without sacrificing generalization performance.
A new model family of zero-inflated Gaussian processes improves prediction and interpretability of rare event data.
problem Poor performance of conventional machine learning on zero-inflated datasets.
method Sparse kernels and latent probit Gaussian processes to zero out kernel rows and columns.
result Improves prediction of zero-inflated data and interpretability of latent mixing models.
New combinatorial method for sparse PCA works beyond spiked identity model.
problem Sparse PCA under general covariance matrices.
method Combinatorial truncated power method with global convergence guarantee.
result First combinatorial sparse PCA method provably successful for general covariance matrices.
Dynamic sparseness reduces neural network computation by selectively omitting parts of computations.
problem Reducing the computational and memory footprint of neural networks.
method Combining dynamic sparseness with block-wise matrix-vector multiplications to selectively omit parts of computations.
result The proposed method outperforms static sparseness and achieves similar perplexities at half the computational cost.
E3 combines sparse MoEs and ensembles to improve model efficiency and performance.
problem Improving model efficiency and performance in machine learning.
method Combining sparse mixture of experts and ensembles, presenting Efficient Ensemble of Experts (E3). result E3 achieves better accuracy, log-likelihood, few-shot learning, robustness, and uncertainty estimates than baselines. SLIMs hybridize sparse linear and isotonic models for high-dimensional data.
problem High-dimensional data with non-linear relationships.
method Hybridizing sparse linear models and isotonic models, proposing a two-step algorithm for estimation.
result The algorithm accurately estimates sparse parameters and monotone functions.
Deep sparse coding models resist adversarial examples.
problem Adversarial examples can fool deep learning models.
method Used deep sparse coding models to resist adversarial examples.
result Deep sparse coding models are robust to adversarial examples.
Sparse Transformers can approximate dense Transformers with only O(n) connections.
problem Can sparse Transformers approximate arbitrary sequence-to-sequence functions?
method Proposed sufficient conditions for universal approximation and proved that sparse Transformers with O(n) connections can approximate dense models.
result Sparse Transformers with O(n) connections can approximate the same function class as dense models with n^2 connections.
A method connects KDE to sparse mixture models with adaptive regularization.
problem Estimating Gaussian mixture models from sparse data.
method Generalized expectation-maximization method with adaptive regularization.
result Sparse mixture models retain details from adaptive KDE.
Paper tackles NAS problem by modeling it as a sparse supernet.
problem Neural Architecture Search (NAS) problem, particularly Mixed-Path Search.
method Model NAS as a sparse supernet with sparsity constraints. Use hierarchical accelerated proximal gradient algorithm for optimization.
result Proposed method finds compact, general, and powerful neural architectures.
Sparse group Lasso optimizes sparse and grouped parameters in high-dimensional data.
problem Simultaneously sparse and grouped parameters in high-dimensional linear regression.
method Sparse group Lasso, debiased sparse group Lasso, statistical inference.
result Matching upper and lower bounds on sample complexity and estimation error.
We consider the problem of modeling multivariate time series with parsimonious dynamical models which can be represented as sparse dynamic Bayesian networks with few latent nodes. This structure translates into a sparse plus low rank model. In this paper, we propose a Gaussian regression approach to identify such a mod…
Sparse neural networks can match dense models on Lipschitz functions.
problem Sparse networks are more efficient but lack theoretical guarantees.
method Formal model of sparse networks, LSH-based routing function, Lipschitz function approximation.
result Sparse networks can approximate dense networks on Lipschitz functions.
New method speeds up sparse graph neural networks training on dense hardware.
problem Training sparse graph neural networks is slow on custom hardware.
method Inspired by sparse matrix optimization, developed techniques for dense hardware.
result Sparse graph neural networks trained in 13 minutes on 512-core TPUv2 Pod.
New estimators improve sparse semiparametric additive modeling.
problem Sparse semiparametric additive modeling with structured sparsity.
method Combines group subset selection with shrinkage for nonconvex optimization.
result New estimators outperform alternatives in synthetic and real-world data.
Simple linear model outperforms deep models on sparse data.
problem Sparse data in recommender systems.
method Linear autoencoder with closed-form solution.
result Simple model achieves better ranking accuracy than deep models.
Entropy regularization improves sparse model discovery in federated learning.
problem Sparse model discovery in federated learning with limited data.
method Entropy regularization of gate distributions for probabilistic sparse model exploration.
result Entropy regularization leads to better sparse model recovery and performance.
New methods improve deep learning models for sparse, high-dimensional data.
problem Underfitting in inference networks for sparse, high-dimensional data.
method Iterative optimization inspired by stochastic variational inference and improvements in sparse data representation.
result State-of-the-art results on text-count dataset and excellent recommendation results.
Block-sparse RNNs reduce model size and improve hardware efficiency.
problem Sparse RNNs do not fully exploit the speed-up potential of sparse operations.
method Two approaches: pruning blocks of weights and using group lasso regularization.
result Block-sparse RNNs achieve 80-90% sparsity with minimal accuracy loss.
New GPU kernels boost deep learning speed and memory efficiency.
problem Sparse deep learning matrices are not well-suited for existing sparse kernels.
method Identified favorable properties of sparse matrices from deep learning, developed high-performance GPU kernels for sparse matrix operations.
result 27% of single-precision peak performance on Nvidia V100 GPUs achieved with new kernels.
Proposes a sparse classifier for discriminative Gaussian Mixture Models.
problem Softmax-based discriminative models assume unimodality, leading to parameter redundancy.
method Sparse Bayesian learning for GMM-based discriminative model, reducing parameters and complexity.
result The SDGM outperforms existing softmax-based discriminative models.