arXiv research

Spectral graph sparsification preserves geometry of GNN embeddings.

FastGAT reduces GNN computation time by 10x using graph sparsification.

Empirical error estimates improve graph sparsification reliability.

This paper improves GNN efficiency for large-scale graph applications.

Spectral sparsification improves Gaussian graphical models under MTP2 constraints.

Spectral sparsification improves Laplacian-constrained graph learning.

While the harmonic function solution performs well in many semi-supervised learning (SSL) tasks, it is known to scale poorly with the number of samples. Recent successful and scalable methods, such as the eigenfunction method focus on efficiently approximating the whole spectrum of the graph Laplacian constructed from …

Proposes a GNN for multivariate time-series prediction with filtering.

Graphs from features improve classification accuracy in tasks.

Proposes a method to improve GNN predictions by finding the most predictive subgraph.

The paper develops a method to sparsify magnetic Laplacians using multi-type spanning forests.

The eigendeomposition of nearest-neighbor (NN) graph Laplacian matrices is the main computational bottleneck in spectral clustering. In this work, we introduce a highly-scalable, spectrum-preserving graph sparsification algorithm that enables to build ultra-sparse NN (u-NN) graphs with guaranteed preservation of the or…

Proposes an offline active learning method for graphs.

A new method efficiently approximates Gromov-Wasserstein distance.

Graphs can model interactions between vertices, but how well depends on graph structure.

S-VNNs improve VNNs by sparsifying covariance matrices.

We study epidemic forecasting on real-world health data by a graph-structured recurrent neural network (GSRNN). We achieve state-of-the-art forecasting accuracy on the benchmark CDC dataset. To improve model efficiency, we sparsify the network weights via transformed-$\ell_1$ penalty and maintain prediction accuracy at…

We derive a new proof to show that the incremental resparsification algorithm proposed by Kelner and Levin (2013) produces a spectral sparsifier in high probability. We rigorously take into account the dependencies across subsequent resparsifications using martingale inequalities, fixing a flaw in the original analysis…

Unified framework sparsifies GNNs for faster inference on large graphs.

Estimating temporal patterns in travel times along road segments in urban settings is of central importance to traffic engineers and city planners. In this work, we propose a methodology to leverage coarse-grained and aggregated travel time data to estimate the street-level travel times of a given metropolitan area. Ou…

Gradient sparsification enhances privacy-preserving machine learning models.

Can one reduce the size of a graph without significantly altering its basic properties? The graph reduction problem is hereby approached from the perspective of restricted spectral approximation, a modification of the spectral similarity measure used for graph sparsification. This choice is motivated by the observation…

Many machine learning frameworks, such as resource-allocating networks, kernel-based methods, Gaussian processes, and radial-basis-function networks, require a sparsification scheme in order to address the online learning paradigm. For this purpose, several online sparsification criteria have been proposed to restrict …

We consider a fundamental algorithmic question in spectral graph theory: Compute a spectral sparsifier of random-walk matrix-polynomial $L_α(G)=D-\sum_{r=1}^dα_rD(D^{-1}A)^r$ where $A$ is the adjacency matrix of a weighted, undirected graph, $D$ is the diagonal matrix of weighted degrees, and $α=(α_1...α_d)$ are nonn…

New method sparsifies hybrid neural ODEs for better performance and stability.

A new federated learning framework with sparsification and adaptive optimization for privacy and efficiency.

We show implicit filter level sparsity manifests in convolutional neural networks (CNNs) which employ Batch Normalization and ReLU activation, and are trained with adaptive gradient descent techniques and L2 regularization or weight decay. Through an extensive empirical study (Mehta et al., 2019) we hypothesize the mec…

Distributed model training suffers from communication overheads due to frequent gradient updates transmitted between compute nodes. To mitigate these overheads, several studies propose the use of sparsified stochastic gradients. We argue that these are facets of a general sparsification method that can operate on any p…

Large scale machine learning is increasingly relying on distributed optimization, whereby several machines contribute to the training process of a statistical model. In this work we study the performance of asynchronous, distributed settings, when applying sparsification, a technique used to reduce communication overhe…

2D-PT improves sampling in constrained optimization problems.

Bayesian sparsification reduces deep neural network complexity.

This paper proposes a method to reveal task relationships in multi-task learning models using sparse graphs.

Deep neural networks have significantly alleviated the burden of feature engineering, but comparable efforts are now required to determine effective architectures for these networks. Furthermore, as network sizes have become excessively large, a substantial amount of resources is invested in reducing their sizes. These…

Modern large scale machine learning applications require stochastic optimization algorithms to be implemented on distributed computational architectures. A key bottleneck is the communication overhead for exchanging information such as stochastic gradients among different workers. In this paper, to reduce the communica…

Learning meaningful graphs from data plays important roles in many data mining and machine learning tasks, such as data representation and analysis, dimension reduction, data clustering, and visualization, etc. In this work, for the first time, we present a highly-scalable spectral approach (GRASPEL) for learning large…

Recently, a lot of techniques were developed to sparsify the weights of neural networks and to remove networks' structure units, e.g. neurons. We adjust the existing sparsification approaches to the gated recurrent architectures. Specifically, in addition to the sparsification of weights and neurons, we propose sparsif…

Efficiently sparsifies simplicial complexes using local densities of states.

New sparsification technique for SGD reduces communication costs.

In graph neural networks (GNNs), pooling operators compute local summaries of input graphs to capture their global properties, and they are fundamental for building deep GNNs that learn hierarchical representations. In this work, we propose the Node Decimation Pooling (NDP), a pooling operator for GNNs that generates c…

New algorithm improves distributed SGD with random sparsification for better convergence and generalization.

In this paper, we investigate effective sketching schemes via sparsification for high dimensional multilinear arrays or tensors. More specifically, we propose a novel tensor sparsification algorithm that retains a subset of the entries of a tensor in a judicious way, and prove that it can attain a given level of approx…

Distributed stochastic gradient descent (SGD) algorithms are widely deployed in training large-scale deep learning models, while the communication overhead among workers becomes the new system bottleneck. Recently proposed gradient sparsification techniques, especially Top-$k$ sparsification with error compensation (To…

Bayesian sparsification improves complex-valued neural networks by 50-100x with minimal performance loss.

Bayesian approach sparsifies neural networks efficiently.

New sparsification theorem for Gaussian processes reduces dimensionality.

New architectures improve KANs, making them more interpretable and accurate.

New regularization techniques improve stability of deep neural networks.

Graph-based semi-supervised learning for aspect term extraction.