A new sparse benchmark metabench identifies key abilities from large benchmarks.
problem Redundancy and compression in existing benchmarks.
method Data from 5000+ LLMs to identify most informative items, distilling a sparse benchmark.
result Sparse benchmark metabench captures underlying abilities with high accuracy.
Improved Compressed Sensing by optimizing sparse solutions with mixed integer programming.
problem Finding sparse solutions to linear measurements with numerical tolerance.
method Introducing an ℓ2 regularized formulation, reformulating as a mixed integer second order cone program, deriving a second order cone relaxation, and developing a custom branch-and-bound algorithm. result Our approach produces solutions that are on average 6.22% more sparse compared to state-of-the-art methods.
This study benchmarks AI agents for personalized retail promotions using simulations.
problem Optimizing coupon targeting for sparse customer purchase events.
method Comprehensive simulations of customer shopping behaviors; training RL agents on batch data.
result Contextual bandit and deep RL methods outperform static policies in sparse reward environments.
It has recently been observed that certain extremely simple feature encoding techniques are able to achieve state of the art performance on several standard image classification benchmarks including deep belief networks, convolutional nets, factored RBMs, mcRBMs, convolutional RBMs, sparse autoencoders and several othe…
Simpler GNNs with low-rank non-parametric aggregators perform well on graph benchmarks.
problem Over-engineering in GNN architectures for common semi-supervised node classification datasets.
method Replacing feature aggregation with a non-parametric learner to streamline GNN design.
result Non-parametric regression is effective for semi-supervised learning on sparse, directed networks.
We introduce Graph-Sparse Logistic Regression, a new algorithm for classification for the case in which the support should be sparse but connected on a graph. We val- idate this algorithm against synthetic data and benchmark it against L1-regularized Logistic Regression. We then explore our technique in the bioinformat…
msPCA solves sparse PCA for multiple components efficiently.
problem Sparse principal component analysis with multiple components.
method Alternating maximization algorithm for sparse loading vectors, with orthogonality or zero correlation constraints.
result Achieves high variance explained with sparse components and controlled feasibility violations.
Architectures for sparse hierarchical representation learning have recently been proposed for graph-structured data, but so far assume the absence of edge features in the graph. We close this gap and propose a method to pool graphs with edge features, inspired by the hierarchical nature of chemistry. In particular, we …
New method for sparse kernel selection improves prediction accuracy.
problem Sparse Multiple Kernel Learning for binary classification.
method Alternating best response algorithm with semidefinite relaxations.
result Method outperforms state-of-the-art MKL approaches in prediction accuracy.
This work follows the approach of multi-label classification for non-intrusive load monitoring (NILM). We modify the popular sparse representation based classification (SRC) approach (developed for single label classification) to solve multi-label classification problems. Results on benchmark REDD and Pecan Street data…
Motivated by an important insight from neural science, we propose a new framework for understanding the success of the recently proposed "maxout" networks. The framework is based on encoding information on sparse pathways and recognizing the correct pathway at inference time. Elaborating further on this insight, we pro…
A new model CDTM improves text classification by concentrating document topics.
problem Unsupervised text classification with diverse topic distributions.
method Imposes an exponential entropy penalty on document topic distribution to encourage concentration.
result More coherent topics and concentrated, sparse document-topic distributions.
This paper optimizes PCE for efficient surrogate modeling in engineering.
problem Efficiently selecting polynomial regressors for surrogate modeling in computationally expensive models.
method Three state-of-the-art basis-adaptive sparse PCE methods are compared and analyzed.
result Automatic selection of the best solver and basis-adaptive scheme improves surrogate model accuracy.
We propose a new method for learning word representations using hierarchical regularization in sparse coding inspired by the linguistic study of word meanings. We show an efficient learning algorithm based on stochastic proximal methods that is significantly faster than previous approaches, making it possible to perfor…
A new EM framework for goal-conditioned RL improves performance on sparse reward tasks.
problem Handling sparse rewards in goal-conditioned reinforcement learning.
method A graphical model framework with an EM algorithm that includes a learning-in-hindsight E-step and a supervised M-step.
result hEM significantly outperforms model-free baselines on goal-conditioned benchmarks with sparse rewards.
We are interested in the decomposition of motion data into a sparse linear combination of base functions which enable efficient data processing. We combine two prominent frameworks: dynamic time warping (DTW), which offers particularly successful pairwise motion data comparison, and sparse coding (SC), which enables an…
In this paper we propose an algorithm that builds sparse decision DAGs (directed acyclic graphs) from a list of base classifiers provided by an external learning method such as AdaBoost. The basic idea is to cast the DAG design task as a Markov decision process. Each instance can decide to use or to skip each base clas…
Topological data analysis (TDA) has emerged as one of the most promising techniques to reconstruct the unknown shapes of high-dimensional spaces from observed data samples. TDA, thus, yields key shape descriptors in the form of persistent topological features that can be used for any supervised or unsupervised learning…
A new hashing method improves accuracy by learning an orthogonal transform.
problem Improving hashing accuracy for similarity search.
method Procrustean Orthogonal Sparse Hashing (POSH) method.
result POSH outperforms state-of-the-art hashing methods.
FF layers in transformers are nearly as interpretable as sparse autoencoders.
problem Comparing interpretability of feature vectors in FF layers vs. sparse autoencoders.
method Revisited interpretability of FF layers as key-value memories using modern benchmarks.
result FF and SAE feature vectors are similarly interpretable, but FFs can be better in some aspects.
Random Fourier features model reconstructs wind fields from sparse measurements.
problem Reconstructing wind fields from limited data.
method Random Fourier features approximating velocity field with adaptive sampling.
result Random Fourier features model outperforms benchmarks.
We study inference and learning based on a sparse coding model with `spike-and-slab' prior. As in standard sparse coding, the model used assumes independent latent sources that linearly combine to generate data points. However, instead of using a standard sparse prior such as a Laplace distribution, we study the applic…
Framework learns continuous dynamics from sparse trajectories.
problem Learning dynamics from sparsely sampled and high-dimensional trajectories.
method Interpolative Multi-Marginal Flow Matching (IMMFM) framework.
result IMMFM outperforms existing methods in forecasting and downstream tasks.
We study parameter estimation in Nonlinear Factor Analysis (NFA) where the generative model is parameterized by a deep neural network. Recent work has focused on learning such models using inference (or recognition) networks; we identify a crucial problem when modeling large, sparse, high-dimensional datasets -- underf…
Neural networks are increasingly used for graph classification in a variety of contexts. Social media is a critical application area in this space, however the characteristics of social media graphs differ from those seen in most popular benchmark datasets. Social networks tend to be large and sparse, while benchmarks …
E3 combines sparse MoEs and ensembles to improve model efficiency and performance.
problem Improving model efficiency and performance in machine learning.
method Combining sparse mixture of experts and ensembles, presenting Efficient Ensemble of Experts (E3). result E3 achieves better accuracy, log-likelihood, few-shot learning, robustness, and uncertainty estimates than baselines. We develop a sparse representation method for neural network uncertainty.
problem Estimating model uncertainty in neural networks.
method Sparse representation of model uncertainty using inverse Multivariate Normal Distribution (MND), with a novel sparsification algorithm and analytical sampler.
result The information form of neural networks can be effectively applied for model uncertainty representation, showing competitive performance.
Entropy regularization improves sparse model discovery in federated learning.
problem Sparse model discovery in federated learning with limited data.
method Entropy regularization of gate distributions for probabilistic sparse model exploration.
result Entropy regularization leads to better sparse model recovery and performance.
A new framework compresses neural networks using sparse optimization.
problem Efficiently reducing the size of deep neural networks for practical deployment.
method Sparse optimization for model compression, tailored for stochastic learning.
result Up to 7.2 and 2.9 times FLOPs reduction with comparable accuracy.
Transfer learning improves sparse, interpretable probabilistic classification.
problem Sparse and interpretable models in transfer learning.
method Two transfer learning extensions integrated into sparse and interpretable probabilistic classification vector machine.
result Transfer learning extensions improve sparsity and performance.
We describe a novel way of representing a symbolic knowledge base (KB) called a sparse-matrix reified KB. This representation enables neural modules that are fully differentiable, faithful to the original semantics of the KB, expressive enough to model multi-hop inferences, and scalable enough to use with realistically…
We consider the problem of portfolio selection within the classical Markowitz mean-variance framework, reformulated as a constrained least-squares regression problem. We propose to add to the objective function a penalty proportional to the sum of the absolute values of the portfolio weights. This penalty regularizes (…
We tackle anomaly detection in sparse time series data.
problem Sparse time series with low signal-to-noise ratios and non-uniform performance.
method We introduce a novel generative procedure for benchmark datasets and demonstrate how anomaly score smoothing improves performance.
result Anomaly score smoothing consistently improves performance in low-count time series anomaly detection.
New method reduces SBL complexity from cubic to linear, improving scalability.
problem Sparse Bayesian Learning's high computational complexity for large feature spaces.
method DQN-SBL, a diagonal Quasi-Newton method for SBL.
result DQN-SBL achieves competitive generalization with sparse models, scaling well to large-scale problems.
Method converts neural networks to function space for better uncertainty quantification.
problem Lack of uncertainty estimates and difficulty in incorporating new data in deep neural networks.
method Dual parameterization to convert from weight space to function space, enabling sparse representation.
result Compact and principled way to capture uncertainty and incorporate new data.
TRUST improves tree models' accuracy while maintaining interpretability.
problem Piecewise-constant regression trees lack in predictive accuracy compared to black-box models.
method Combines Random Forest accuracy with interpretability of shallow trees and sparsity of linear models, using LLMs for explanations.
result TRUST outperforms other interpretable models in predictive accuracy and matches Random Forest's accuracy.
A new knot selection method speeds up sparse Gaussian process approximations.
problem Efficiently selecting knots for sparse Gaussian processes.
method One-at-a-time Bayesian optimization for knot selection.
result Competitive performance with reduced computational cost.
Sparse prototypes improve clustering of high-dimensional directional data.
problem Clustering high-dimensional directional data like texts.
method Estimate a von Mises mixture using l1 penalized likelihood and EM algorithm.
result Sparse prototypes enhance interpretability and clustering performance.
Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them. But also generic hardware and software implementations of deep learning run more efficiently for sparse networks. Several methods exist for pruning connections of a neural network after it was trained without conne…
This paper begins with considering the identification of sparse linear time-invariant networks described by multivariable ARX models. Such models possess relatively simple structure thus used as a benchmark to promote further research. With identifiability of the network guaranteed, this paper presents an identificatio…
SSVI efficiently trains sparse Bayesian neural networks with minimal compression and performance loss.
problem Efficiently training Bayesian neural networks with uncertainty quantification.
method SSVI optimizes a sparse subspace basis selection and its parameters alternately, guided by weight distribution statistics.
result SSVI achieves significant compression (10-20x model size reduction) with minimal performance drop (under 3%) and FLOPs reduction (up to 20x) compared to dense Variational Inference.
The implementation of conventional sparse principal component analysis (SPCA) on high-dimensional data sets has become a time consuming work. In this paper, a series of subspace projections are constructed efficiently by using Household QR factorization. With the aid of these subspace projections, a fast deflation meth…
Paper solves k-sparse parity problem with sign SGD, matching SQ lower bound.
problem Solving k-sparse parity problems efficiently.
method Sign stochastic gradient descent on neural networks.
result Matches Statistical Query lower bound for solving k-sparse parity problems.
ETGL-DDPG improves DDPG for sparse reward control with new exploration and replay techniques.
problem Sparse reward continuous control in reinforcement learning.
method Introduces εt-greedy search and GDRB framework for efficient exploration and reward use. result ETGL-DDPG outperforms DDPG and other methods on sparse-reward continuous benchmarks.
This work shows how disentangled and sparse representations improve multi-task learning.
problem Improving generalization in multi-task learning with disentangled and sparse representations.
method Proved a new identifiability result and proposed a practical approach using sparsity-promoting bi-level optimization.
result Maximally sparse base-predictors yield disentangled representations under certain conditions.
Model learns disentangled representations from natural videos.
problem Disentangling factors of variation in natural data.
method Sparse prior on temporally adjacent observations.
result Model reliably learns disentangled representations on natural data.
Sparse hypergraph neural networks improve reasoning in large knowledge graphs.
problem Reasoning about relationships in large, real-world domains using sparse and local inferences.
method Sparse and local hypergraph neural networks (SpaLoc) exploiting relational inferences that are usually local and sparse.
result State-of-the-art performance on real-world knowledge graph reasoning benchmarks.
SRMD uses random features for efficient time-frequency analysis.
problem Efficiently analyzing time-series data with low computational cost.
method Sparse Random Mode Decomposition (SRMD) constructs a sparse approximation to the spectrogram.
result SRMD outperforms other methods in signal representation, outlier removal, and mode decomposition.