A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Law derived for neural networks with sparse connections.
USN improves neural networks with uniform sparse connectivity.
Sparse Transformers can approximate dense Transformers with only O(n) connections.
We propose Sparse Neural Network architectures that are based on random or structured bipartite graph topologies. Sparse architectures provide compression of the models learned and speed-ups of computations, they can also surpass their unstructured or fully connected counterparts. As we show, even more compact topologi…
This work introduces a method to compare sparse neural network topologies using graph theory.
Neuromorphic hardware tends to pose limits on the connectivity of deep networks that one can run on them. But also generic hardware and software implementations of deep learning run more efficiently for sparse networks. Several methods exist for pruning connections of a neural network after it was trained without conne…
Transformers excel at sparse token selection, surpassing FCNs in both worst and average cases.
Sparse connectivity improves generalization in neural networks below the Edge of Stability.
We present a transductive deep learning-based formulation for the sparse representation-based classification (SRC) method. The proposed network consists of a convolutional autoencoder along with a fully-connected layer. The role of the autoencoder network is to learn robust deep features for classification. On the othe…
Paper connects neural network hyperparameter optimization and NAS to structured sparse recovery.
Clarifies connections between Nyström and SVGP methods for scalable GPs.
We propose a novel technique to assess functional brain connectivity in EEG/MEG signals. Our method, called Sparsely-Connected Sources Analysis (SCSA), can overcome the problem of volume conduction by modeling neural data innovatively with the following ingredients: (a) the EEG is assumed to be a linear mixture of corr…
Many transformations in deep learning architectures are sparsely connected. When such transformations cannot be designed by hand, they can be learned, even through plain backpropagation, for instance in attention mechanisms. However, during learning, such sparse structures are often represented in a dense form, as we d…
This paper addresses the problem of inferring sparse causal networks modeled by multivariate auto-regressive (MAR) processes. Conditions are derived under which the Group Lasso (gLasso) procedure consistently estimates sparse network structure. The key condition involves a "false connection score." In particular, we sh…
New algorithms reduce communication in GNN training.
Recurrent neural networks (RNNs) are omnipresent in sequence modeling tasks. Practical models usually consist of several layers of hundreds or thousands of neurons which are fully connected. This places a heavy computational and memory burden on hardware, restricting adoption in practical low-cost and low-power devices…
Improvements in the performance of deep neural networks have often come through the design of larger and more complex networks. As a result, fast memory is a significant limiting factor in our ability to improve network performance. One approach to overcoming this limit is the design of sparse neural networks, which ca…
We discuss the systemic risk implied by the interbank exposures reconstructed with the maximum entropy method. The maximum entropy method severely underestimates the risk of interbank contagion by assuming a fully connected network, while in reality the structure of the interbank network is sparsely connected. Here, we…
We propose a novel denoising framework for task functional Magnetic Resonance Imaging (tfMRI) data to delineate the high-resolution spatial pattern of the brain functional connectivity via dictionary learning and sparse coding (DLSC). In order to address the limitations of the unsupervised DLSC-based fMRI studies, we u…
New pruning methods improve dynamic sparse training performance.
Analyzes geodesic lengths in sparse networks, deriving a distribution.
Stochastic Sparse Subspace Clustering improves subspace clustering by reducing over-segmentation through dropout.
Proposes RBGP framework for efficient block sparse neural networks.
This paper considers the growth in the length of one-dimensional trajectories as they are passed through deep ReLU neural networks, which, among other things, is one measure of the expressivity of deep networks. We generalise existing results, providing an alternative, simpler method for lower bounding expected traject…
Framework for inferring latent structure from sparse, imperfectly detected bipartite networks.
A simple 2-layer linear network outperforms neural networks in learning sparse targets.
Sparse Subspace Clustering (SSC) is one of the most popular methods for clustering data points into their underlying subspaces. However, SSC may suffer from heavy computational burden. Orthogonal Matching Pursuit applied on SSC accelerates the computation but the trade-off is the loss of clustering accuracy. In this pa…
Sparse Hopfield model improves memory retrieval with fewer connections.
We propose a framework to model the distribution of sequential data coming from a set of entities connected in a graph with a known topology. The method is based on a mixture of shared hidden Markov models (HMMs), which are jointly trained in order to exploit the knowledge of the graph structure and in such a way that …
We introduce Graph-Sparse Logistic Regression, a new algorithm for classification for the case in which the support should be sparse but connected on a graph. We val- idate this algorithm against synthetic data and benchmark it against L1-regularized Logistic Regression. We then explore our technique in the bioinformat…
This paper introduces structure learning for autoencoder recommenders to improve performance and generalization.
Functional brain networks are well described and estimated from data with Gaussian Graphical Models (GGMs), e.g. using sparse inverse covariance estimators. Comparing functional connectivity of subjects in two populations calls for comparing these estimated GGMs. Our goal is to identify differences in GGMs known to hav…
Sparse symmetric tensor regression reduces brain connectivity complexity.
The paper shows that g-convex functions on manifolds are sparse.
New method finds sparse networks without labels, improving performance.
Study on signal recovery from low-rank matrix with sparse noise.
LiPopt uses polynomial optimization to estimate neural network Lipschitz constants efficiently.
We provide novel guaranteed approaches for training feedforward neural networks with sparse connectivity. We leverage on the techniques developed previously for learning linear networks and show that they can also be effectively adopted to learn non-linear networks. We operate on the moments involving label and the sco…
New principle for disentangling latent factors using sparse regularization.
SpaceNet improves continual learning by intelligently compressing neural connections.
Is it possible to find the sparsest vector (direction) in a generic subspace with ? This problem can be considered a homogeneous variant of the sparse recovery problem, and finds connections to sparse dictionary learning, sparse PCA, and many other …
The recently proposed Multi-Layer Convolutional Sparse Coding (ML-CSC) model, consisting of a cascade of convolutional sparse layers, provides a new interpretation of Convolutional Neural Networks (CNNs). Under this framework, the computation of the forward pass in a CNN is equivalent to a pursuit algorithm aiming to e…
This paper proposes a method to reveal task relationships in multi-task learning models using sparse graphs.
New methods solve sparse estimation robustly, even with outliers.
A new method selects features efficiently for high-dimensional data.
Subspace clustering is the problem of clustering data points into a union of low-dimensional linear/affine subspaces. It is the mathematical abstraction of many important problems in computer vision, image processing and machine learning. A line of recent work (4, 19, 24, 20) provided strong theoretical guarantee for s…
Paper connects probability density cuts to graph theory eigenfunctions.
NGRs merge sparse graph recovery with PGMs for efficient probabilistic inference.