Sparse neural encoding can store more memories as targets become sparser.
problem Storing sparse input-target associations in neural networks.
method Mathematical proofs using properties of random polytopes and sub-gaussian random vector variables.
result The capacity of neural maps increases with sparsity in target layers.
New method uses neural maps to efficiently sample lattice QCD distributions.
problem Challenges in sampling Boltzmann distributions of lattice field theories.
method Sparse triangular transport maps exploiting conditional independence structure of lattice graphs.
result Sparse triangular maps achieve efficient sampling with linear time complexity in lattice size.
Graph neural networks improve AMG convergence for sparse systems.
problem Efficiently constructing algebraic multigrid prolongation operators for sparse linear systems.
method Train a graph neural network to learn prolongation operators from matrix classes, using an unsupervised loss function.
result Improved convergence rates compared to classical AMG methods.
Olshausen and Field (OF) proposed that neural computations in the primary visual cortex (V1) can be partially modeled by sparse dictionary learning. By minimizing the regularized representation error they derived an online algorithm, which learns Gabor-filter receptive fields from a natural image ensemble in agreement …
A new estimator learns sparse linear models with context-dependent coefficients.
problem Sparse linear models lack flexibility compared to deep neural networks for handling feature groups.
method Contextual lasso estimator using a deep neural network with lasso regularization.
result Learned models can be sparser than standard lasso without sacrificing predictive power.
VolNP learns IVS from sparse quotes via meta-learning and SABR priors.
problem Reconstructing implied volatility surfaces from sparse option quotes.
method Meta-learning Neural Process with SABR-induced priors.
result VolNP outperforms SABR, SSVI, and Gaussian process on SPX options.
DistGP models multi-robot mapping with distributed Gaussian process learning.
problem Collaborative mapping by multiple robots with limited local data.
method Sparse Gaussian process with factorisation and distributed training via GBP.
result DistGP achieves superior accuracy and robustness compared to DiNNO.
Sparse neural networks can match dense models on Lipschitz functions.
problem Sparse networks are more efficient but lack theoretical guarantees.
method Formal model of sparse networks, LSH-based routing function, Lipschitz function approximation.
result Sparse networks can approximate dense networks on Lipschitz functions.
Structured prediction requires searching over a combinatorial number of structures. To tackle it, we introduce SparseMAP: a new method for sparse structured inference, and its natural loss function. SparseMAP automatically selects only a few global structures: it is situated between MAP inference, which picks a single …
New model estimates sparse transport maps for high-dimensional data.
problem Estimating optimal transport maps in high-dimensional spaces.
method Proposes a new model using a family of translation invariant costs and sparsity-inducing norms.
result Sparse transport maps that apply Occam's razor to reduce complexity.
Proposes HBGNN for better recommendation systems using graph neural networks.
problem Sparse structured data in recommendation systems lacking feature richness.
method Hierarchical BiGraph Neural Network (HBGNN) using bigraph framework.
result Competitive performance compared to current methods.
Proposes a graph neural network for efficient multi-agent routing.
problem Routing multiple agents in complex, sparsely connected graphs with dynamic traffic.
method Value Iteration Network based on graph neural networks for online coordination.
result Significantly outperforms traditional methods in terms of cost and runtime.
In compressed sensing, we wish to reconstruct a sparse signal x from observed data y. In sparse coding, on the other hand, we wish to find a representation of an observed signal y as a sparse linear combination, with coefficients x, of elements from an overcomplete dictionary. While many algorithms are competit…
Graph matching with feature vectors is solved using a two-layer graph neural network.
problem Graph matching in the presence of sparse binary features.
method Two-layer graph neural network with graph structure.
result Graph neural network can recover correct mapping with high probability under certain conditions.
Many data-driven approaches exist to extract neural representations of functional magnetic resonance imaging (fMRI) data, but most of them lack a proper probabilistic formulation. We propose a group level scalable probabilistic sparse factor analysis (psFA) allowing spatially sparse maps, component pruning using automa…
This work introduces a method to compare sparse neural network topologies using graph theory.
problem Comparing and understanding sparse neural network topologies, especially during training.
method Introducing Neural Network Sparse Topology Distance (NNSTD) to measure distances between different sparse neural networks.
result Sparse neural networks can outperform over-parameterized models without further structure optimization.
This paper improves the scalability of sparse neural network compression.
problem Sparse neural network compression for diverse data modalities.
method State-of-the-art sparsification techniques and meta-learning.
result Meta-learning sparse compression networks achieve new state-of-the-art results.
New method for efficient marginalization of discrete latent variables in neural networks.
problem Computational challenges in training models with discrete latent variables.
method Parameterizing discrete distributions using sparse mappings (sparsemax and structured variants) to reduce support and enable efficient marginalization.
result Achieved good performance in various tasks with efficient and practical training.
Sparse deep neural networks(DNNs) are efficient in both memory and compute when compared to dense DNNs. But due to irregularity in computation of sparse DNNs, their efficiencies are much lower than that of dense DNNs on regular parallel hardware such as TPU. This inefficiency leads to poor/no performance benefits for s…
New method finds sparse networks without labels, improving performance.
problem Sparse connectivity in neural networks to reduce memory and energy demands.
method Neural Tangent Transfer method to find sparse networks without labels.
result Sparse networks achieve higher classification performance and faster convergence.
Paper tackles NAS problem by modeling it as a sparse supernet.
problem Neural Architecture Search (NAS) problem, particularly Mixed-Path Search.
method Model NAS as a sparse supernet with sparsity constraints. Use hierarchical accelerated proximal gradient algorithm for optimization.
result Proposed method finds compact, general, and powerful neural architectures.
This paper explores loss landscapes of sparse neural networks, finding unique characteristics compared to dense networks.
problem Understanding the loss landscape of sparse neural networks, especially one-hidden-layer networks.
method Analyzes sparse networks with dense and sparse final layers, focusing on linear and non-linear models.
result Sparse networks can have no spurious valleys under certain conditions, but spurious valleys and minima can exist for wide sparse networks.
Law derived for neural networks with sparse connections.
problem Understanding the behavior of neural networks with sparse connections.
method Law of large numbers for empirical distribution of parameters derived.
result Law for neural networks with sparse connections derived.
Guarantees sparse recovery for neural networks with iterative hard thresholding.
problem Recovering sparse network weights in neural networks.
method Structural properties of sparse network weights and iterative hard thresholding algorithm.
result Simple iterative hard thresholding algorithm recovers sparse network weights exactly using linear memory.
The paper tackles learning varying DAG structures based on contextual features.
problem Learning a single DAG for the entire population from observational data.
method A neural network that maps contextual features to a weighted adjacency matrix of a DAG, with a projection layer to ensure acyclicity.
result The new approach can recover context-specific DAGs where existing methods fail.
New property helps SGD learn sparse functions efficiently in neural networks.
problem Characterizing functions learnable by SGD in non-linear neural networks.
method Mean-field analysis, hierarchical merged-staircase property, dimension-free dynamics approximation.
result Merged-staircase property is necessary and nearly sufficient for SGD learnability.
Oracle inequality for sparse neural nets adapts to unknown structure.
problem Sparse deep neural nets in nonparametric regression.
method Gibbs posterior distribution with Metropolis-adjusted Langevin algorithms and mixture of uniform priors.
result Oracle inequality showing adaptation to unknown regularity and structure, achieving minimax-optimal rate of convergence.
We consider machine learning techniques to develop low-latency approximate solutions to a class of inverse problems. More precisely, we use a probabilistic approach for the problem of recovering sparse stochastic signals that are members of the ℓp-balls. In this context, we analyze the Bayesian mean-square-error …
A fast method combines deep mixtures of sparse GPs for flexible modeling.
problem Flexible modeling with changing output densities.
method Designing gating network with DNN for selecting sparse GPs, using CCR algorithm.
result The method outperforms competing methods in accuracy and uncertainty quantification.
The sizes of deep neural networks (DNNs) are rapidly outgrowing the capacity of hardware to store and train them. Research over the past few decades has explored the prospect of sparsifying DNNs before, during, and after training by pruning edges from the underlying topology. The resulting neural network is known as a …
Sparse-penalized deep neural networks improve performance in weakly dependent processes.
problem Nonparametric regression and classification under weak dependence.
method Sparse-penalized deep neural networks with oracle inequalities and convergence rates established.
result The proposed estimators outperform non-penalized ones in simulations.
Renormalized pruning improves neural network accuracy.
problem Over-parameterized neural networks waste many parameters.
method Propose renormalizing sparse neural networks.
result Renormalized pruning converges to zero error.
Sparse linear models improve neural network debuggability.
problem Improving neural network interpretability and debugging.
method Using sparse linear models over learned deep feature representations.
result The approach leads to more debuggable and accurate neural networks.
USN improves neural networks with uniform sparse connectivity.
problem Overfitting and limited scalability in classical neural networks.
method Uniform sparse network (USN) with even and sparse connectivity.
result USN outperforms state-of-the-art sparse network models in accuracy, speed, and robustness.
GrateTile optimizes CNN feature map storage for efficient data access.
problem Efficient storage and access of sparse CNN feature maps.
method Divides feature maps into uneven-sized subtensors, compresses and stores them in a compressed yet accessible format.
result Average 55% DRAM bandwidth reduction with minimal indexing overhead.
Graph neural networks have become increasingly popular in recent years due to their ability to naturally encode relational input data and their ability to scale to large graphs by operating on a sparse representation of graph adjacency matrices. As we look to scale up these models using custom hardware, a natural assum…
The sizes of deep neural networks (DNNs) are rapidly outgrowing the capacity of hardware to store and train them. Research over the past few decades has explored the prospect of sparsifying DNNs before, during, and after training by pruning edges from the underlying topology. The resulting neural network is known as a …
Paper connects neural network hyperparameter optimization and NAS to structured sparse recovery.
problem Hyperparameter optimization and neural architecture search in neural networks.
method Structured sparse recovery methods applied to HPO and NAS.
result Improvements in hyperparameter optimization and discovery of novel neural architectures.
Modern neural networks are often augmented with an attention mechanism, which tells the network where to focus within the input. We propose in this paper a new framework for sparse and structured attention, building upon a smoothed max operator. We show that the gradient of this operator defines a mapping from real val…
New method uses sparse deep neural networks for high-dimensional regression with improved parameter estimation.
problem Improving parameter estimation in high-dimensional sparse regression models.
method Proposes nonparametric estimation of partial derivatives in sparse deep neural networks.
result Established convergence rate of nonparametric estimation of partial derivatives as O(n−1/4). We propose Sparse Neural Network architectures that are based on random or structured bipartite graph topologies. Sparse architectures provide compression of the models learned and speed-ups of computations, they can also surpass their unstructured or fully connected counterparts. As we show, even more compact topologi…
Learning to approximate a separable function is hard, requiring many samples even with sparse networks.
problem Learning the separable function x↦∑i=1dxi2 with limited samples. method Sparse neural networks vs. dense neural networks, explicit regularization.
result The sample complexity for dense networks is O(d2.5) with explicit regularization, better than O(d4). Paper improves DOA estimation in sparse arrays using Siamese neural networks.
problem Challenges in DOA estimation with limited snapshots in sparse linear arrays.
method Introduces a Siamese neural network with a sparse augmentation layer for enhanced signal feature embedding.
result Demonstrates improved DOA estimation accuracy in sparse arrays.
Random projections help in representing sparse graphs efficiently.
problem Efficiently representing sparse graphs of varying sizes and vertex sets.
method Random projection of adjacency matrices to retain graph functionality and properties.
result Random projections can accurately represent graphs of different sizes and vertex sets in the same space.
New framework tackles deep learning issues like local traps and miscalibration.
problem Local traps and miscalibration in deep neural networks.
method Sparse deep learning framework with prior annealing algorithms.
result Proposed method successfully addresses local traps and miscalibration.
This paper explores efficient neural networks by identifying sparse structures.
problem Reducing computational complexity in neural networks without sacrificing accuracy.
method Identifying and utilizing sparse structures in neural network weights and activations.
result Large and sparse models are more beneficial for practical problems.
Hybrid Bayesian neural networks use function uncertainty for probabilistic inference.
problem Uncertainty in neural network weights is hard to specify and interpret.
method Integrates probabilistic layers with standard deterministic layers for function uncertainty.
result Improves probabilistic inference by encoding function uncertainty.
We consider the classical sparse regression problem of recovering a sparse signal x0 given a measurement vector y=Φx0+w. We propose a tree search algorithm driven by the deep neural network for sparse regression (TSN). TSN improves the signal reconstruction performance of the deep neural network designed for sp…