Zero imputation leads to suboptimal neural network performance.
problem Suboptimal performance in neural networks due to missing data.
method Introduced Sparsity Normalization (SN) to handle missingness.
result Debiasing input-level sparsity improves neural network performance and stability.
Deep learning can learn compositional functions more efficiently by breaking them into stages.
problem Understanding why deep learning performs better than shallow models in learning compositional functions.
method Analyzed learnability of compositional target functions using a three-layer fitting model trained with layer-wise spectral estimators.
result Learning compositional functions can be simplified by breaking them into stages, reducing the complexity of the learning problem.
ISAAC audits deep models for drug-target interactions, revealing structural differences.
problem Deep models for DTI often use irrelevant features, making them hard to evaluate.
method ISAAC uses intervention-based structural auditing to evaluate model sensitivity.
result ISAAC reveals significant structural differences in DTI models' reasoning.
New method discovers concepts in hidden feature layers using sparse subspace clustering.
problem Local attribution methods fail to identify coherent model behavior across samples.
method Sparse Subspace Clustering (SSCC) for concept discovery.
result Empirically validated method for various image classification tasks.
A new framework detects concept drift in streaming data.
problem Detecting distributional changes in non-stationary data streams.
method Treating model parameters as random variables, ERICS uses information theory measures to identify concept drift.
result ERICS effectively detects concept drift compared to existing methods.
MixDiff detects OOD samples in constrained access environments by comparing perturbed samples.
problem Detecting out-of-distribution samples in models with restricted access.
method Apply identical perturbation to target and similar ID sample, compare model outputs.
result MixDiff enhances OOD detection performance consistently across various datasets.
New layer sparsity concept improves neural networks.
problem Improving neural network efficiency and interpretability.
method Formulated layer sparsity, introduced regularization and refitting schemes.
result Generated more compact and accurate neural networks.
Our work is focused on the joint sparsity recovery problem where the common sparsity pattern is corrupted by Poisson noise. We formulate the confidence-constrained optimization problem in both least squares (LS) and maximum likelihood (ML) frameworks and study the conditions for perfect reconstruction of the original r…
STR reparameterizes DNN weights with soft thresholds for better sparsity and accuracy.
problem Improving sparsity in DNNs for better accuracy and lower inference cost.
method Soft Threshold Reparameterization (STR) using the soft-threshold operator on DNN weights.
result STR achieves state-of-the-art accuracy and reduces FLOPs by up to 50%.
Coarse-grained pruning improves sparsity efficiency without sacrificing accuracy.
problem Efficiency of hardware design and prediction accuracy in sparse CNNs.
method Quantitative analysis of sparsity regularity vs. accuracy trade-off.
result Coarse-grained pruning achieves similar sparsity ratios and accuracy as fine-grained pruning.
New theorem for generalized group sparsity improves consistency and convergence rates.
problem Improving statistical inference in high-dimensional data with element-wise and group-wise sparsity.
method Developed a generalized version of Sparse-Group Lasso and proved a universal theorem for consistency and convergence rates.
result Obtained results on consistency and convergence rates for different forms of double sparsity regularization.
This work proposes an attention mechanism for structured sparsity in neural networks.
problem Uncontrolled sparsity in neural networks leads to significant accuracy drops.
method An attention mechanism that controls sparsity intensity and network pruning.
result The proposed method outperforms baseline methods by 6% on CIFAR-10.
This paper studies activation sparsity in large language models, finding key trends and implications.
problem Activation sparsity in large language models (LLMs) can be improved for efficiency and interpretability.
method Proposes PPL-p% sparsity, analyzes trends with training data, width-depth ratio, and parameter scale. result ReLU is more efficient for sparsity than SiLU, and deeper architectures can improve sparsity.
New method targets sparsity to prevent overfitting in deep nets.
problem Overfitting in deep neural networks with small datasets.
method Targeted sparsity regularization to visualize and counteract overfitting.
result Significant increase in image classification performance without overfitting.
Embeds sparsity in deep neural networks, allowing exact zero parameters.
problem Learning sparse structures in deep networks.
method Embeds sparsity into neural network structure, allowing exact zero parameters during training.
result Can learn both structured and unstructured sparsity.
Social-sparsity brain decoders improve speed and interpretability.
problem Computational cost and interpretability in brain decoding models.
method Introduced social-sparsity, a structured shrinkage operator.
result Social-sparsity performs almost as well as total-variation models and better than graph-net, with a fraction of the computational cost.
New sparsity attacks degrade DNN efficiency, raising concerns for resource-constrained systems.
problem Vulnerabilities in DNNs through energy and latency attacks.
method Proposed sparsity attacks that modify DNN inputs to reduce activation sparsity, increasing execution time and energy consumption.
result Adversarial sparsity attacks can degrade DNN efficiency by up to 1.82x in image recognition DNNs.
Sparseout controls sparsity in neural networks, improving performance in language modeling.
problem Controlling sparsity in neural networks to optimize performance.
method Sparseout is a variant of Dropout that controls sparsity, theoretically proven and empirically validated.
result Sparseout controls the desired level of sparsity in neural network activations, improving performance in language modeling.
Complexity helps identify sparse risk factors in asset pricing.
problem Tension between feature richness and economic parsimony in high-dimensional asset pricing.
method Expanding feature space and using basis pursuit to discover sparse risk factors.
result Nonlinear feature expansions combined with basis pursuit yield superior out-of-sample performance.
Spark Transformer achieves high sparsity in FFN and attention without sacrificing model quality.
problem Inactivity of most neurons in FFN of Transformers, leading to inefficiency.
method Top-k masking, statistical top-k algorithm, and predictor for activated entries.
result Only 8% of FFN neurons are activated, achieving up to 2.5x wall-time speedup.
New method recovers block-sparse signals with common sparsity patterns.
problem Recovering block-sparse signals with common sparsity patterns in MMV.
method Pattern-coupled hierarchical Gaussian prior model with EM framework.
result Proposed method automatically captures block sparse structure.
The paper investigates sparsity in deep neural networks, especially in larger models.
problem Reducing resource demands in deep neural networks, particularly in larger models.
method Extended TensorQuant toolbox to investigate sparsity in deeper models and various classification problem sizes.
result Promoted sparsity in deeper models, showing differences in sparsity for activations, weights, and gradients.
New method for multiclass classification reduces error bounds.
problem High-dimensional multiclass classification with sparse coefficients.
method Sparse multinomial logistic regression with convex penalties.
result Plug-in classifiers achieve minimax generalization error bounds.
New method enforces encoder sparsity in HPF for more interpretable feature selection.
problem Lack of encoder sparsity in HPF leads to lack of column-clustering property.
method Enforces encoder sparsity using a generalized additive model (GAM).
result Gains ability to perform feature selection and relates each representation to original features.
Unified framework for generalized sparsity and RIP analysis.
problem Analyzing inverse problems with sparsity models.
method Proposed generalized notions of sparsity and a unified RIP framework.
result Extends RIP analysis to broader contexts including tensor products.
The paper studies how regularization parameters affect sparsity in deep neural networks.
problem Reducing the complexity of deep neural networks by promoting sparsity.
method Derives ℓ1-norm sparsity-promoting models, characterizes sparsity levels, and develops algorithms for selecting optimal regularization parameters. result Developed algorithms to select regularization parameters for desired sparsity levels in neural networks.
Paper proposes efficient algorithm for recovering sparsity pattern from deterministic missing data.
problem Recovering sparsity pattern from datasets with deterministic missing structure.
method Proposes an efficient algorithm for missing value imputation using topological property of censorship filter.
result Consistently recovers the sparsity pattern with high probability in polynomial time and logarithmic sample complexity.
New method solves graph-structured sparsity problems efficiently.
problem Graph-structured sparsity optimization in complex models.
method Stochastic gradient-based approach for non-convex graph-structured sparsity.
result Linear convergence up to a constant error.
Recent results in Compressive Sensing have shown that, under certain conditions, the solution to an underdetermined system of linear equations with sparsity-based regularization can be accurately recovered by solving convex relaxations of the original problem. In this work, we present a novel primal-dual analysis on a …
Sparsity helps reduce diffusion model costs.
problem High computational costs in diffusion models.
method Introduced sparsity concept to reduce input dimensionality.
result Sparsity reduces computational complexity to intrinsic data dimension.
Study on sparsity in CNNs trained with adaptive methods.
problem Understanding and optimizing sparsity in CNNs trained with adaptive methods.
method Experimental study of filter level sparsity in CNNs with BN and ReLU, using adaptive gradient descent and L2 regularization.
result Implicit sparsity can improve CNN performance and speedup without modifications.
New interactive greedy algorithm for group sparsity in high dimensions.
problem Benefits of group sparsity for greedy-type methods in high-dimensional data analysis.
method Interactive Greedy Approach
result Proposed algorithm attains desired benefits of group sparsity under high dimensional settings.
Efficient algorithm selects atoms from dictionaries with complex sparsity constraints.
problem Dictionary selection with complex sparsity constraints.
method Novel efficient greedy algorithm for dictionary selection.
result Outperforms known methods in faster running time and competitive performance.
New ICA method exploits sparsity for better brain imaging analysis.
problem ICA's independence assumption is too strict for real-world data.
method Entropy bound minimization with sparsity exploitation.
result Improved ICA performance through direct incorporation of sparsity.
SparseRT accelerates sparse computations on GPUs for deep learning inference.
problem Efficiently handling unstructured sparsity patterns on GPUs for deep learning.
method SparseRT, a code generator that leverages unstructured sparsity for accelerating sparse linear algebra operations.
result Geometric mean speedups of 3.4x at 90% sparsity and 5.4x at 95% sparsity for 1x1 convolutions and fully connected layers.
SparseTrain uses dynamic sparsity in training deep neural networks on CPUs.
problem Training deep neural networks efficiently on general-purpose processors.
method Exploits dynamic zeros introduced by ReLU in feature maps and gradients.
result Significantly speeds up training on CPUs, up to 1.51x.
New methods solve graph sparsity optimization problems faster.
problem Complex graph sparsity optimization problems in disease outbreak monitoring and social network analysis.
method Stochastic variance-reduced gradient-based methods GraphSVRG-IHT and GraphSCSG-IHT.
result Our methods achieve linear convergence speed.
Compressive sensing (CS) exploits sparsity to recover sparse or compressible signals from dimensionality reducing, non-adaptive sensing mechanisms. Sparsity is also used to enhance interpretability in machine learning and statistics applications: While the ambient dimension is vast in modern data analysis problems, the…
Sparse activations in neural models correlate with frequent words, suggesting sparsity is natural.
problem Interpretability and resource efficiency in neural language models.
method Used the Taxi-Euclidean norm to measure sparsity and analyzed gradients and activations of frequent words.
result Frequent input words are associated with sparse activations, while frequent target words are associated with dispersed activations.
DeepHoyer introduces differentiable, scale-invariant sparsity measures for neural networks.
problem Efficiently sparsifying neural networks with scale-invariant sparsity measures.
method Developed DeepHoyer, a set of differentiable, scale-invariant sparsity-inducing regularizers based on the Hoyer measure.
result DeepHoyer produces sparser neural networks than previous methods, maintaining similar accuracy.
Improves robustness of information bottleneck framework with sparsity-inducing prior.
problem Fixed-dimensional priors restrict flexibility and restrict robustness.
method Sparsity-inducing spike-slab categorical prior that learns dimension distribution per data point.
result Improves accuracy and robustness compared to traditional priors and other methods.
Two sparsity-aware NSAF algorithms improve sparse system identification with lower complexity.
problem Sparse system identification with improved performance and lower complexity.
method Gradient descent method to minimize combined cost function and l1-norm penalty on filter coefficients.
result Proposed algorithms achieve comparable performance with lower computational complexity.
New methods generalize nonlinear ICA beyond structural sparsity.
problem Identify true latent sources from nonlinear mixtures without structural sparsity assumptions.
method Propose identifiability results for undercomplete, partial sparsity, and flexible grouping structures.
result Prove identifiability in general settings of undercompleteness, partial sparsity, and flexible grouping structures.
Paper proposes a unified sparsity-based framework for evaluating algorithmic fairness.
problem Ensuring fairness in machine learning across diverse domains.
method Unified sparsity-based framework for evaluating fairness.
result Demonstrates broad applicability and effectiveness of the framework.
Recent studies in the literature have paid much attention to the sparsity in linear classification tasks. One motivation of imposing sparsity assumption on the linear discriminant direction is to rule out the noninformative features, making hardly contribution to the classification problem. Most of those work were focu…
Bit-slice sparsity improves ReRAM-based DNN acceleration.
problem Limited ADC power and area constraints in ReRAM-based DNN accelerators.
method Proposed bit-slice L1 algorithm to induce sparsity during training.
result 2x sparsity improvement compared to previous methods.
This paper investigates the role of sparsity in Reservoir Computing networks.
problem Designing efficient Recurrent Neural Networks (RNNs) with hidden recurrent layers.
method Empirical investigation of sparsity in input-reservoir connections and recurrent connections.
result Sparsity, particularly in input-reservoir connections, enhances the network's temporal memory and dimensionality.
This paper optimizes exclusive sparsity norm minimization with random groupings.
problem Sparse feature selection with even distribution across groups.
method Developed efficient algorithms for exclusive sparsity norm minimization with smooth and non-smooth losses, and proposed random grouping scheme for unknown group information.
result Achieved optimal convergence rate for exclusive sparsity norm minimization.