arXiv research

Paper tackles NAS problem by modeling it as a sparse supernet.

Dynamic Sparse Training finds efficient sparse networks from scratch.

Sparse transformer architecture improves accuracy and speed in generative modeling and inverse problems.

MLP-Mixer achieves better performance through sparsity and wider architecture.

We present a comprehensive framework for structured sparse coding and modeling extending the recent ideas of using learnable fast regressors to approximate exact sparse codes. For this purpose, we develop a novel block-coordinate proximal splitting method for the iterative solution of hierarchical sparse coding problem…

Paper connects neural network hyperparameter optimization and NAS to structured sparse recovery.

Deep learning boosts building energy load forecasting.

Proposes a new framework for predicting stock market movements using sparse neural architectures.

Deep learning models reconstruct volatility surfaces from noisy data under no-arbitrage constraints.

We propose Sparse Neural Network architectures that are based on random or structured bipartite graph topologies. Sparse architectures provide compression of the models learned and speed-ups of computations, they can also surpass their unstructured or fully connected counterparts. As we show, even more compact topologi…

New framework for neural networks converging to low loss without overparameterization.

Many transformations in deep learning architectures are sparsely connected. When such transformations cannot be designed by hand, they can be learned, even through plain backpropagation, for instance in attention mechanisms. However, during learning, such sparse structures are often represented in a dense form, as we d…

Evidential Softmax preserves multimodality in sparse probability distributions for generative models.

Paper hypothesizes MLP layers in LLMs can be approximated by sparse Mixture of Experts.

AdaEnsemble learns adaptive feature interactions for CTR prediction.

Proposes a novel neural architecture for sparse coding using learned greedy pursuit.

FiGS searches over a larger space of architectures for efficient mobile models.

New neural KB representation speeds up reasoning with large symbolic knowledge bases.

Sparse activations in neural networks are hard to exploit but lead to advantages in learning.

Boosts share routing for multi-task learning with flexible sparse connections.

There has been a rapid progress in the task of Visual Question Answering with improved model architectures. Unfortunately, these models are usually computationally intensive due to their sheer size which poses a serious challenge for deployment. We aim to tackle this issue for the specific task of Visual Question Answe…

GrateTile optimizes CNN feature map storage for efficient data access.

Paper reveals hidden convexities in deep learning models using sparse signal processing.

New algorithms improve neural architecture search with faster convergence.

In this paper, we present a new deep learning architecture for addressing the problem of supervised learning with sparse and irregularly sampled multivariate time series. The architecture is based on the use of a semi-parametric interpolation network followed by the application of a prediction network. The interpolatio…

Generates high-quality images using sparse DCT representations.

Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse factorizations of the attention matrix which reduce this to $O(n \sqrt{n})$. We also introduce a) a variation on architecture and initialization to train deeper net…

VINNAS uses variational inference to avoid mode collapse in neural architecture search.

Improved SINDy autoencoder for identifying noisy dynamical systems.

Theory explains why DARTS favors deep architectures over shallow ones.

We investigate the difficulties of training sparse neural networks and make new observations about optimization dynamics and the energy landscape within the sparse regime. Recent work of \citep{Gale2019, Liu2018} has shown that sparse ResNet-50 architectures trained on ImageNet-2012 dataset converge to solutions that a…

Recurrent neural networks (RNNs) are powerful and effective for processing sequential data. However, RNNs are usually considered "black box" models whose internal structure and learned parameters are not interpretable. In this paper, we propose an interpretable RNN based on the sequential iterative soft-thresholding al…

SIVF k-means algorithm speeds up sparse data clustering.

SpaceNet improves continual learning by intelligently compressing neural connections.

EASIER-net uses sparse networks to improve prediction accuracy for high-dimensional data.

Optimal graph classification uses message-passing neural networks.

Hierarchical temporal memory (HTM) is a biomimetic sequence memory algorithm that holds promise for invariant representations of spatial and spatiotemporal inputs. This paper presents a comprehensive neuromemristive crossbar architecture for the spatial pooler (SP) and the sparse distributed representation classifier, …

Paper proposes a new sparse Bayesian neural network for simpler, more efficient DNNs.

New LSTM scheme incorporates prior knowledge and measurement uncertainties.

Transformers excel at sparse token selection, surpassing FCNs in both worst and average cases.

In this paper, we propose a novel recurrent neural network architecture for speech separation. This architecture is constructed by unfolding the iterations of a sequential iterative soft-thresholding algorithm (ISTA) that solves the optimization problem for sparse nonnegative matrix factorization (NMF) of spectrograms.…

Hydra boosts efficiency for long-context reasoning in resource-constrained settings.

Self-attention prefers sparse functions of input sequences, reducing sample complexity.

Paper infers human mobility from sparse trajectories.

This paper presents a new neural architecture that combines a modulated Hebbian network (MOHN) with DQN, which we call modulated Hebbian plus Q network architecture (MOHQA). The hypothesis is that such a combination allows MOHQA to solve difficult partially observable Markov decision process (POMDP) problems which impa…

Capsule Networks improve autonomous navigation in sparse environments.

Introduces top-$k$ regularization for better feature selection in machine learning.

Graph Convolutional Networks (GCNs) have proven to be successful tools for semi-supervised learning on graph-based datasets. For sparse graphs, linear and polynomial filter functions have yielded impressive results. For large non-sparse graphs, however, network training and evaluation becomes prohibitively expensive. B…