New method enforces encoder sparsity in HPF for more interpretable feature selection.
problem Lack of encoder sparsity in HPF leads to lack of column-clustering property.
method Enforces encoder sparsity using a generalized additive model (GAM).
result Gains ability to perform feature selection and relates each representation to original features.
Efficient algorithm selects atoms from dictionaries with complex sparsity constraints.
problem Dictionary selection with complex sparsity constraints.
method Novel efficient greedy algorithm for dictionary selection.
result Outperforms known methods in faster running time and competitive performance.
This paper investigates the role of sparsity in Reservoir Computing networks.
problem Designing efficient Recurrent Neural Networks (RNNs) with hidden recurrent layers.
method Empirical investigation of sparsity in input-reservoir connections and recurrent connections.
result Sparsity, particularly in input-reservoir connections, enhances the network's temporal memory and dimensionality.
New loss function for learning sparse representations from data.
problem Emergence of sparse representations in neural networks.
method Analysis of input data distribution and regularization.
result Introduction of a new loss function for sparse regularization.
Proposes a method to determine the most compact representation in deep neural networks.
problem Finding the most compact representation in deep neural networks.
method Layer-wise pruning method considering layer importance and total target sparsity.
result Achieved up to 75% better top-5 accuracy with 75% compression than baseline.
This work shows how disentangled and sparse representations improve multi-task learning.
problem Improving generalization in multi-task learning with disentangled and sparse representations.
method Proved a new identifiability result and proposed a practical approach using sparsity-promoting bi-level optimization.
result Maximally sparse base-predictors yield disentangled representations under certain conditions.
Improves robustness of information bottleneck framework with sparsity-inducing prior.
problem Fixed-dimensional priors restrict flexibility and restrict robustness.
method Sparsity-inducing spike-slab categorical prior that learns dimension distribution per data point.
result Improves accuracy and robustness compared to traditional priors and other methods.
Many natural signals exhibit a sparse representation, whenever a suitable describing model is given. Here, a linear generative model is considered, where many sparsity-based signal processing techniques rely on such a simplified model. As this model is often unknown for many classes of the signals, we need to select su…
New neural operators learn structured patterns efficiently.
problem Learning and representing complex, structured patterns in data.
method Sparse autoencoder neural operators (SAE-NOs) parameterize concepts as functions, enabling efficient and structured representation.
result SAE-FNOs learn localized patterns and generalize across different scales and discretizations.
Paper tackles selfless sequential learning with neural inhibition to improve future task capacity.
problem Learning tasks in sequence with limited model capacity.
method Study regularization strategies and activation functions, proposing a novel representation sparsity regularizer.
result Representation sparsity regularizer improves performance over alternative regularizers.
A new method for learning network representations that avoids information bias and sparsity.
problem Information bias and sparsity in network representation learning.
method A spreading-activation schema for learning node embeddings in network structures.
result Significant improvement in various real-world network analysis tasks.
Method identifies latent variables from high-dimensional data with piecewise affine mixing.
problem Identifying latent variables from high-dimensional observations with dependencies and piecewise affine transformations.
method Proposes a two-stage method with sparsity and Gaussianity regularization.
result Effectively recovers ground-truth latent variables from synthetic and image data.
We propose a simple and efficient time-series clustering framework particularly suited for low Signal-to-Noise Ratio (SNR), by simultaneous smoothing and dimensionality reduction aimed at preserving clustering information. We extend the sparse K-means algorithm by incorporating structured sparsity, and use it to exploi…
A novel algorithm optimizes sparsity in reservoir computing inspired by insect brain.
problem Optimizing sparsity in reservoir computing networks.
method Inspired by insect brain, the algorithm optimizes sparsity levels by adjusting node firing thresholds.
result The algorithm outperforms standard gradient descent on tasks involving better classification, memorization, and convergence.
Paper improves Tm prediction of protein fragments using sparsity and probabilistic models.
problem Improving accuracy of melting temperature prediction for protein fragments.
method Promoting sparsity in pre-trained transformer models and adopting probabilistic frameworks.
result Mean absolute error of 0.23C for predicting melting temperature.
New method identifies causal variables from partially observed data.
problem Learning from unpaired observations with instance-dependent partial observability.
method Proposes two methods enforcing sparsity in the inferred representation.
result Establishes two identifiability results for linear and piecewise linear mixing functions.
Grassmannian packings improve CNN kernels' diversity and reduce sparsity.
problem Kernel sparsity and lack of diversity in CNNs decrease model capacity.
method Initialize CNN kernels with Grassmannian packings to maximize diversity and minimize sparsity.
result Grassmannian packings lead to diverse features and improved classification accuracy.
DNNs with L2 regularization reveal feature learning dynamics and sparsity.
problem Understanding feature learning in DNNs with L2 regularization. method Reformulating loss in terms of layerwise activations and covariances.
result Proving sparsity of local minima in L2-regularized DNNs. DMGE learns cross-domain user behavior embeddings using multi-graphs and GNNs.
problem Data sparsity in learning large-scale item embedding from individual domain data.
method Construct multi-graphs from users' behaviors across domains, use multi-graph neural networks to learn cross-domain representation.
result DMGE outperforms state-of-the-art embedding methods in various tasks.
SparseTrain uses dynamic sparsity in training deep neural networks on CPUs.
problem Training deep neural networks efficiently on general-purpose processors.
method Exploits dynamic zeros introduced by ReLU in feature maps and gradients.
result Significantly speeds up training on CPUs, up to 1.51x.
We present sparse topical coding (STC), a non-probabilistic formulation of topic models for discovering latent representations of large collections of data. Unlike probabilistic topic models, STC relaxes the normalization constraint of admixture proportions and the constraint of defining a normalized likelihood functio…
This paper investigates learning sparse representations and action-value functions simultaneously in deep reinforcement learning.
problem Mitigating catastrophic interference and improving cumulative reward in deep reinforcement learning agents.
method Employing regularization techniques to learn sparse representations and action-value functions incrementally.
result Learning sparse representations can improve performance and robustness in deep reinforcement learning agents.
Recent breakthrough results in compressed sensing (CS) have established that many high dimensional objects can be accurately recovered from a relatively small number of non- adaptive linear projection observations, provided that the objects possess a sparse representation in some basis. Subsequent efforts have shown th…
Lower class selectivity makes networks more robust to natural perturbations but more vulnerable to adversarial attacks.
problem Understanding how class selectivity affects robustness to different types of perturbations in neural networks.
method Investigated the relationship between class selectivity and robustness to natural and adversarial perturbations in neural networks.
result Lower class selectivity increases robustness to natural perturbations but decreases robustness to adversarial attacks.
Adaptively sparse Transformers improve interpretability and diversity in NLP.
problem Standard Transformers use dense attention, limiting interpretability and diversity.
method Introduces adaptively sparse Transformers using α-entmax for context-dependent sparsity. result Improves interpretability and diversity in NLP tasks without sacrificing accuracy.
The paper develops a multi-kernel method with sparsity constraint for regression.
problem Developing a robust regression method with sparsity constraints.
method Banach-space formulation, generalized total-variation regularization, multi-kernel expansion, adaptive kernel positions, ℓ1 penalty on coefficients. result The method achieves sparsity in the kernel coefficients, reducing the number of active kernels to the number of data points.
BiSHop tackles tabular data challenges with sparse Hopfield layers.
problem Non-rotationally invariant data structure and feature sparsity in tabular data.
method Sequential column-wise and row-wise processing through interconnected directional learning modules with generalized sparse modern Hopfield layers.
result BiSHop surpasses current SOTA methods with significantly less hyperparameter tuning.
We develop a highly scalable optimization method called "hierarchical group-thresholding" for solving a multi-task regression model with complex structured sparsity constraints on both input and output spaces. Despite the recent emergence of several efficient optimization algorithms for tackling complex sparsity-induci…
Sparse estimation methods are aimed at using or obtaining parsimonious representations of data or models. They were first dedicated to linear variable selection but numerous extensions have now emerged such as structured sparsity or kernel selection. It turns out that many of the related estimation problems can be cast…
The paper explores algorithms to transform 3-manifold triangulations while controlling sparsity.
problem Designing efficient algorithms for 3-manifold triangulations with controlled sparsity.
method Revisit and apply a linear-time algorithm for converting triangulations into Heegaard diagrams, and present a quasi-linear-time algorithm for retriangulation.
result Quasi-linear-time algorithm producing a Heegaard diagram with controlled sparsity.
Deep neural networks define suitable reproducing kernel Banach spaces.
problem Characterizing the function spaces of deep neural networks.
method Reproducing kernel Banach spaces and variational results.
result Deep neural networks define suitable reproducing kernel Banach spaces.
A new objective function for NMF reduces model complexity and improves accuracy.
problem NMF's error-based objective function can lead to overly complex models.
method MDL-NMF uses minimum description length to balance model complexity and accuracy.
result MDL-NMF outperforms traditional NMF on various datasets.
SpaRCe optimizes reservoir computing by learning neuron thresholds to improve performance and prevent forgetting.
problem Improving performance and preventing forgetting in reservoir computing networks.
method Integrates neuron-specific learnable thresholds to optimize sparsity without altering dynamics, learning read-out weights and thresholds via gradient rule.
result Threshold learning improves performance and alleviates catastrophic forgetting.
New theory for partial disentanglement from sparse graphs.
problem Disentangling latent factors from sparse causal graphs.
method Generalization of disentanglement theory to any graph, using consistency equivalence.
result Partial disentanglement captures expected factor entanglement based on graph structure.
Paper presents techniques to classify UWB SAR imagery, distinguishing targets from clutter.
problem Distinguishing obscured targets from clutter in UWB SAR imagery.
method Three novel sparsity-driven techniques exploiting tensor coefficients and polarization diversity.
result Tensor sparsity models enhance classification accuracy of multi-channel SAR data.
Two complementary approaches have been extensively used in signal and image processing leading to novel results, the sparse representation methodology and the variational strategy. Recently, a new sparsity based model has been proposed, the cosparse analysis framework, which may potentially help in bridging sparse appr…
We propose to optimize the activation functions of a deep neural network by adding a corresponding functional regularization to the cost function. We justify the use of a second-order total-variation criterion. This allows us to derive a general representer theorem for deep neural networks that makes a direct connectio…
Sparsity inducing regularization is an important part for learning over-complete visual representations. Despite the popularity of ℓ1 regularization, in this paper, we investigate the usage of non-convex regularizations in this problem. Our contribution consists of three parts. First, we propose the leaky capped …
Dictionary learning algorithms have been successfully used in both reconstructive and discriminative tasks, where the input signal is represented by a linear combination of a few dictionary atoms. While these methods are usually developed under ℓ1 sparsity constrain (prior) in the input domain, recent studies hav…
SRHM explains deep learning's hierarchy and insensitivity to transformations.
problem Understanding how deep networks learn hierarchical and invariant representations.
method Introducing sparsity to generative hierarchical models of data.
result Hierarchical representations and insensitivity to transformations correlate strongly with deep network performance.
Model combines user preferences and side information for better recommendation.
problem Addressing data sparsity and improving user intent representation.
method Tensor-based model that fuses user preferences with side information.
result Demonstrates effectiveness on standard benchmark datasets.
This paper tackles causal representation learning from multiple distributions without hard interventions.
problem Recovering latent causal variables and their relations from multiple distributions.
method Develops general solutions for causal representation learning without hard interventions, under sparsity constraints and suitable change conditions.
result Recovering the moralized graph of the underlying directed acyclic graph and latent variables related to the underlying causal model.
Study examines how statistical properties of deep learning representations can be adjusted.
problem Improving performance in deep learning models.
method Investigated eight representation regularization methods, including two new rank regularizers.
result Manipulating statistical properties of representations can indirectly improve model performance.
New model allows sparse graphs with many triangles to be represented.
problem Sparse graphs with many triangles cannot be accurately represented in finite dimensions.
method Infinite-dimensional inner product model with manifold representations.
result Local neighborhoods can be represented in lower dimensions.
Study confirms sparse coding in whole brain using MRI data.
problem Sparse coding in the whole brain's neural activities.
method Applied various matrix factorization methods to fMRI data.
result Sparse coding hypothesis in information representation in the whole human brain is confirmed.
Most existing approaches address multi-view subspace clustering problem by constructing the affinity matrix on each view separately and afterwards propose how to extend spectral clustering algorithm to handle multi-view data. This paper presents an approach to multi-view subspace clustering that learns a joint subspace…
Diffusion MRI (dMRI) provides the ability to reconstruct neuronal fibers in the brain, in vivo, by measuring water diffusion along angular gradient directions in q-space. High angular resolution diffusion imaging (HARDI) can produce better estimates of fiber orientation than the popularly used diffusion tens…
The paper analyzes the trade-off between smoothness and sparsity in GCN using lp-regularized learning.
problem Quantifying the trade-off between smoothness and sparsity in GCN.
method Proposes a novel SGD proximal algorithm for GCNs with an inexact operator to analyze the stability of the ℓp-regularized stochastic learning. result Establishes an explicit theoretical understanding of GCN with ℓp-regularized stochastic learning.