A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
This paper studies activation sparsity in large language models, finding key trends and implications.
Sparsity helps reduce diffusion model costs.
Our work is focused on the joint sparsity recovery problem where the common sparsity pattern is corrupted by Poisson noise. We formulate the confidence-constrained optimization problem in both least squares (LS) and maximum likelihood (ML) frameworks and study the conditions for perfect reconstruction of the original r…
Embeds sparsity in deep neural networks, allowing exact zero parameters.
The paper studies how regularization parameters affect sparsity in deep neural networks.
Spark Transformer achieves high sparsity in FFN and attention without sacrificing model quality.
Spatially-sparse predictors are good models for brain decoding: they give accurate predictions and their weight maps are interpretable as they focus on a small number of regions. However, the state of the art, based on total variation or graph-net, is computationally costly. Here we introduce sparsity in the local neig…
Compressive sensing (CS) exploits sparsity to recover sparse or compressible signals from dimensionality reducing, non-adaptive sensing mechanisms. Sparsity is also used to enhance interpretability in machine learning and statistics applications: While the ambient dimension is vast in modern data analysis problems, the…
Dropout is commonly used to help reduce overfitting in deep neural networks. Sparsity is a potentially important property of neural networks, but is not explicitly controlled by Dropout-based regularization. In this work, we propose Sparseout a simple and efficient variant of Dropout that can be used to control the spa…
New method enforces encoder sparsity in HPF for more interpretable feature selection.
STR reparameterizes DNN weights with soft thresholds for better sparsity and accuracy.
New method targets sparsity to prevent overfitting in deep nets.
Many natural signals exhibit a sparse representation, whenever a suitable describing model is given. Here, a linear generative model is considered, where many sparsity-based signal processing techniques rely on such a simplified model. As this model is often unknown for many classes of the signals, we need to select su…
New methods solve graph sparsity optimization problems faster.
Paper proposes a new sparsity scheme for high-dimensional VAR models.
Paper studies binary random projections with controllable sparsity patterns for computational and accuracy advantages.
New measure SEV shows non-sparse models can still have low decision sparsity.
Working in high-dimensional latent spaces, the internal encoding of data in Variational Autoencoders becomes naturally sparse. We discuss this known but controversial phenomenon sometimes refereed to as overpruning, to emphasize the under-use of the model capacity. In fact, it is an important form of self-regularizatio…
In this paper, we investigate a new compressive sensing model for multi-channel sparse data where each channel can be represented as a hierarchical tree and different channels are highly correlated. Therefore, the full data could follow the forest structure and we call this property as \emph{forest sparsity}. It exploi…
Study finds economic data may not be as sparse as previously thought.
Adversarial robustness improved by sparsity in network weights.
Stochastic optimization algorithms update models with cheap per-iteration costs sequentially, which makes them amenable for large-scale data analysis. Such algorithms have been widely studied for structured sparse models where the sparsity information is very specific, e.g., convex sparsity-inducing norms or -n…
New Bayesian method for joint sparse parameter inference.
MuVI models multi-view data with structured sparsity, integrating domain knowledge.
In the present paper we study a sparse stochastic network enabled with a block structure. The popular Stochastic Block Model (SBM) and the Degree Corrected Block Model (DCBM) address sparsity by placing an upper bound on the maximum probability of connections between any pair of nodes. As a result, sparsity describes o…
Paper introduces structured sparsity estimators for Generalized Linear Models.
HALO learns to prune neural networks by adaptively shrinking weights.
We propose a new sparsity-smoothness penalty for high-dimensional generalized additive models. The combination of sparsity and smoothness is crucial for mathematical theory as well as performance for finite-sample data. We present a computationally efficient algorithm, with provable numerical convergence properties, fo…
New layer sparsity concept improves neural networks.
Bayesian framework for encoding uncertainty and inducing sparsity.
Study the effects of data parallelism and sparsity on neural network training.
Sparsity-promoting priors have become increasingly popular over recent years due to an increased number of regression and classification applications involving a large number of predictors. In time series applications where observations are collected over time, it is often unrealistic to assume that the underlying spar…
Paper proposes a unified sparsity-based framework for evaluating algorithmic fairness.
In this paper, we consider the block-sparse signals recovery problem in the context of multiple measurement vectors (MMV) with common row sparsity patterns. We develop a new method for recovery of common row sparsity MMV signals, where a pattern-coupled hierarchical Gaussian prior model is introduced to characterize bo…
Deep learning is finding its way into the embedded world with applications such as autonomous driving, smart sensors and aug- mented reality. However, the computation of deep neural networks is demanding in energy, compute power and memory. Various approaches have been investigated to reduce the necessary resources, on…
This paper describes a simple framework for structured sparse recovery based on convex optimization. We show that many structured sparsity models can be naturally represented by linear matrix inequalities on the support of the unknown parameters, where the constraint matrix has a totally unimodular (TU) structure. For …
LEWIS merges LLMs without training, improving performance on specific tasks.
Improves robustness of information bottleneck framework with sparsity-inducing prior.
Proposes a neural network for sparsity regularization in inverse problems using Gaussian mixture.
This work introduces adversarial sparsity to measure robustness beyond adversarial accuracy.
Concerns about interpretability, computational resources, and principled inductive priors have motivated efforts to engineer sparse neural models for NLP tasks. If sparsity is important for NLP, might well-trained neural models naturally become roughly sparse? Using the Taxi-Euclidean norm to measure sparsity, we find …
Sparsity-constrained optimization is an important and challenging problem that has wide applicability in data mining, machine learning, and statistics. In this paper, we focus on sparsity-constrained optimization in cases where the cost function is a general nonlinear function and, in particular, the sparsity constrain…
New Max-Plus neural network exploits subgradient sparsity for efficient training.
The restricted isometry property (RIP) is an integral tool in the analysis of various inverse problems with sparsity models. Motivated by the applications of compressed sensing and dimensionality reduction of low-rank tensors, we propose generalized notions of sparsity and provide a unified framework for the correspond…
In seeking for sparse and efficient neural network models, many previous works investigated on enforcing L1 or L0 regularizers to encourage weight sparsity during training. The L0 regularizer measures the parameter sparsity directly and is invariant to the scaling of parameter values, but it cannot provide useful gradi…
New theorem for generalized group sparsity improves consistency and convergence rates.
DSA efficiently allocates sparsity across layers for budgeted pruning.
Automated PDE discovery from multiple noisy experiments.