arXiv research

New sparsity attacks degrade DNN efficiency, raising concerns for resource-constrained systems.

New algorithm speeds up regression tasks for large datasets.

SparseTrain uses dynamic sparsity in training deep neural networks on CPUs.

Algorithm learns latent simplex from perturbed points in input-sparsity time.

This paper investigates the role of sparsity in Reservoir Computing networks.

Rocket algorithm classifies time-series data efficiently using random projections and natural sparsity.

While variational dropout approaches have been shown to be effective for network sparsification, they are still suboptimal in the sense that they set the dropout rate for each neuron without consideration of the input data. With such input-independent dropout, each neuron is evolved to be generic across inputs, which m…

Within machine learning, the supervised learning field aims at modeling the input-output relationship of a system, from past observations of its behavior. Decision trees characterize the input-output relationship through a series of nested $if-then-else$ questions, the testing nodes, leading to a set of predictions, th…

New method finds sparse groups of input variables for neural networks.

Improved Frank-Wolfe algorithm speeds up training of differentially private LASSO models.

Total least squares regression sped up with input sparsity time.

A new data-oblivious sketch for logistic regression reduces data size while maintaining approximation accuracy.

Sparsity helps reduce diffusion model costs.

SpinSVAR estimates SVAR models with sparse input, improving accuracy and scalability.

NARD extends ARD for linear models, promoting sparsity and correlation structure.

New algorithm reduces matrix multiplication time for sparse matrices.

To improve the execution speed and efficiency of neural networks in embedded systems, it is crucial to decrease the model size and computational complexity. In addition to conventional compression techniques, e.g., weight pruning and quantization, removing unimportant activations can reduce the amount of data communica…

Neural networks have been shown to be vulnerable against minor adversarial perturbations of their inputs, especially for high dimensional data under $\ell_\infty$ attacks. To combat this problem, techniques like adversarial training have been employed to obtain models which are robust on the training set. However, the …

Handling missing data is one of the most fundamental problems in machine learning. Among many approaches, the simplest and most intuitive way is zero imputation, which treats the value of a missing entry simply as zero. However, many studies have experimentally confirmed that zero imputation results in suboptimal perfo…

Dictionary learning algorithms have been successfully used in both reconstructive and discriminative tasks, where the input signal is represented by a linear combination of a few dictionary atoms. While these methods are usually developed under $\ell_1$ sparsity constrain (prior) in the input domain, recent studies hav…

Concerns about interpretability, computational resources, and principled inductive priors have motivated efforts to engineer sparse neural models for NLP tasks. If sparsity is important for NLP, might well-trained neural models naturally become roughly sparse? Using the Taxi-Euclidean norm to measure sparsity, we find …

HS-MoE selects sparse experts using adaptive priors and data-adaptive gating.

Transformers exhibit sparse activation maps, reducing computational load and improving robustness.

Working in high-dimensional latent spaces, the internal encoding of data in Variational Autoencoders becomes naturally sparse. We discuss this known but controversial phenomenon sometimes refereed to as overpruning, to emphasize the under-use of the model capacity. In fact, it is an important form of self-regularizatio…

DIET-SNN optimizes SNNs for faster, lower-energy image classification.

Efficient method certifies robustness of discrete data models, especially graphs.

Recurrent neural networks (RNNs) have drawn interest from machine learning researchers because of their effectiveness at preserving past inputs for time-varying data processing tasks. To understand the success and limitations of RNNs, it is critical that we advance our analysis of their fundamental memory properties. W…

2L-FUSE enhances feature sparsity through kernel learning.

First-order stochastic methods are the state-of-the-art in large-scale machine learning optimization owing to efficient per-iteration complexity. Second-order methods, while able to provide faster convergence, have been much less explored due to the high cost of computing the second-order information. In this paper we …

Improves robustness of information bottleneck framework with sparsity-inducing prior.

Reactmine infers chemical reactions from time series data, overcoming sparse model limitations.

We study the emergence of sparse representations in neural networks. We show that in unsupervised models with regularization, the emergence of sparsity is the result of the input data samples being distributed along highly non-linear or discontinuous manifold. We also derive a similar argument for discriminatively trai…

The paper proposes a control strategy for systems with sparse parameters using compressed sensing.

CoT improves transformer sample efficiency by reducing input token dependencies and attention sparsity.

We propose two sparsity-aware normalized subband adaptive filter (NSAF) algorithms by using the gradient descent method to minimize a combination of the original NSAF cost function and the l1-norm penalty function on the filter coefficients. This l1-norm penalty exploits the sparsity of a system in the coefficients upd…

We compress large neural networks for quick adaptation to specific contexts.

New algorithm learns PTFs with noisy data efficiently.

TATD predicts missing entries in time-evolving tensors by exploiting temporal dependency and sparsity.

This work introduces adversarial sparsity to measure robustness beyond adversarial accuracy.

A new method for the unsupervised learning of sparse representations using autoencoders is proposed and implemented by ordering the output of the hidden units by their activation value and progressively reconstructing the input in this order. This can be done efficiently in parallel with the use of cumulative sums and …

Proposes a sparsity algorithm to improve corporate credit ratings.

DNNs with $L_{2}$ regularization reveal feature learning dynamics and sparsity.

Spark Transformer achieves high sparsity in FFN and attention without sacrificing model quality.

Paper proposes a new sparsity scheme for high-dimensional VAR models.

We develop a highly scalable optimization method called "hierarchical group-thresholding" for solving a multi-task regression model with complex structured sparsity constraints on both input and output spaces. Despite the recent emergence of several efficient optimization algorithms for tackling complex sparsity-induci…

Given two sets of variables, derived from a common set of samples, sparse Canonical Correlation Analysis (CCA) seeks linear combinations of a small number of variables in each set, such that the induced canonical variables are maximally correlated. Sparse CCA is NP-hard. We propose a novel combinatorial algorithm for s…

BayTiDe discovers time-delayed differential equations from noisy data.

This work assesses DNNs for estimating conditional probabilities.