A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
This paper explores loss landscapes of sparse neural networks, finding unique characteristics compared to dense networks.
This work introduces a method to compare sparse neural network topologies using graph theory.
Dynamic Sparse Training finds efficient sparse networks from scratch.
New method finds sparse networks without labels, improving performance.
Guarantees sparse recovery for neural networks with iterative hard thresholding.
Sparse deep neural networks(DNNs) are efficient in both memory and compute when compared to dense DNNs. But due to irregularity in computation of sparse DNNs, their efficiencies are much lower than that of dense DNNs on regular parallel hardware such as TPU. This inefficiency leads to poor/no performance benefits for s…
Sparse neural networks can match dense models on Lipschitz functions.
Law derived for neural networks with sparse connections.
This paper considers the growth in the length of one-dimensional trajectories as they are passed through deep ReLU neural networks, which, among other things, is one measure of the expressivity of deep networks. We generalise existing results, providing an alternative, simpler method for lower bounding expected traject…
Sparse linear models improve neural network debuggability.
USN improves neural networks with uniform sparse connectivity.
SnAp approximates RTRL for online training of sparse recurrent networks.
We propose a method for solving statistical mechanics problems defined on sparse graphs. It extracts a small Feedback Vertex Set (FVS) from the sparse graph, converting the sparse system to a much smaller system with many-body and dense interactions with an effective energy on every configuration of the FVS, then learn…
Most artificial networks today rely on dense representations, whereas biological networks rely on sparse representations. In this paper we show how sparse representations can be more robust to noise and interference, as long as the underlying dimensionality is sufficiently high. A key intuition that we develop is that …
Renormalized pruning improves neural network accuracy.
We consider the classical sparse regression problem of recovering a sparse signal given a measurement vector . We propose a tree search algorithm driven by the deep neural network for sparse regression (TSN). TSN improves the signal reconstruction performance of the deep neural network designed for sp…
Analyzes geodesic lengths in sparse networks, deriving a distribution.
Improvements in the performance of deep neural networks have often come through the design of larger and more complex networks. As a result, fast memory is a significant limiting factor in our ability to improve network performance. One approach to overcoming this limit is the design of sparse neural networks, which ca…
The sizes of deep neural networks (DNNs) are rapidly outgrowing the capacity of hardware to store and train them. Research over the past few decades has explored the prospect of sparsifying DNNs before, during, and after training by pruning edges from the underlying topology. The resulting neural network is known as a …
We present a transductive deep learning-based formulation for the sparse representation-based classification (SRC) method. The proposed network consists of a convolutional autoencoder along with a fully-connected layer. The role of the autoencoder network is to learn robust deep features for classification. On the othe…
We demonstrate the possibility of what we call sparse learning: accelerated training of deep neural networks that maintain sparse weights throughout training while achieving dense performance levels. We accomplish this by developing sparse momentum, an algorithm which uses exponentially smoothed gradients (momentum) to…
Sparse-penalized deep neural networks improve performance in weakly dependent processes.
Many applications require sparse neural networks due to space or inference time restrictions. There is a large body of work on training dense networks to yield sparse networks for inference, but this limits the size of the largest trainable sparse model to that of the largest trainable dense model. In this paper we int…
Graph neural networks have become increasingly popular in recent years due to their ability to naturally encode relational input data and their ability to scale to large graphs by operating on a sparse representation of graph adjacency matrices. As we look to scale up these models using custom hardware, a natural assum…
The sizes of deep neural networks (DNNs) are rapidly outgrowing the capacity of hardware to store and train them. Research over the past few decades has explored the prospect of sparsifying DNNs before, during, and after training by pruning edges from the underlying topology. The resulting neural network is known as a …
Graph neural networks improve AMG convergence for sparse systems.
Sparse activations in neural networks are hard to exploit but lead to advantages in learning.
This paper begins with considering the identification of sparse linear time-invariant networks described by multivariable ARX models. Such models possess relatively simple structure thus used as a benchmark to promote further research. With identifiability of the network guaranteed, this paper presents an identificatio…
Recent work on deep neural network pruning has shown there exist sparse subnetworks that achieve equal or improved accuracy, training time, and loss using fewer network parameters when compared to their dense counterparts. Orthogonal to pruning literature, deep neural networks are known to be susceptible to adversarial…
The paper provides statistical guarantees for sparse deep learning.
A new framework compresses neural networks using sparse optimization.
Sparse spectral decomposition identifies overlapping communities in networks.
New method uses sparse deep neural networks for high-dimensional regression with improved parameter estimation.
New CRM models for sparse networks with linear edge growth.
This paper explores efficient neural networks by identifying sparse structures.
Proposes a method to learn sparse deep neural networks with theoretical guarantees.
Paper improves DOA estimation in sparse arrays using Siamese neural networks.
We propose Sparse Neural Network architectures that are based on random or structured bipartite graph topologies. Sparse architectures provide compression of the models learned and speed-ups of computations, they can also surpass their unstructured or fully connected counterparts. As we show, even more compact topologi…
New framework tackles deep learning issues like local traps and miscalibration.
Sparse hypergraph neural networks improve reasoning in large knowledge graphs.
Networks are a unifying framework for modeling complex systems and network inference problems are frequently encountered in many fields. Here, I develop and apply a generative approach to network inference (RCweb) for the case when the network is sparse and the latent (not observed) variables affect the observed ones. …
Paper develops efficient variational inference for sparse deep learning with theoretical guarantees.
FedDST trains sparse sub-networks to improve efficiency in federated learning.
Sparse codes improve optimal control tasks with correlated inputs.
Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them. But also generic hardware and software implementations of deep learning run more efficiently for sparse networks. Several methods exist for pruning connections of a neural network after it was trained without conne…
Paper proposes an efficient algorithm for learning sparse Bayesian networks from discrete high-dimensional data.
Paper solves k-sparse parity problem with sign SGD, matching SQ lower bound.
Generative model captures hubs and dense communities in social networks.