arXiv research

New technique debiases distributed optimization, improving convergence rate.

Newton-LESS sparsifies Gaussian sketching for faster optimization.

New methods for sketching non-PSD matrices improve regression and optimization tasks.

Considering the existence of very large amount of available data repositories and reach to the very advanced system of hardware, systems meant for facial identification ave evolved enormously over the past few decades. Sketch recognition is one of the most important areas that have evolved as an integral component adop…

We address the statistical and optimization impacts of the classical sketch and Hessian sketch used to approximately solve the Matrix Ridge Regression (MRR) problem. Prior research has quantified the effects of classical sketch on the strictly simpler least squares regression (LSR) problem. We establish that classical …

Paper proposes a new method for density estimation using tree tensor-network states.

New method reduces linear regret in high-dimensional bandit problems.

Develops precise expressions for random projections for better machine learning tasks.

Sketching techniques have become popular for scaling up machine learning algorithms by reducing the sample size or dimensionality of massive data sets, while still maintaining the statistical power of big data. In this paper, we study sketching from an optimization point of view: we first show that the iterative Hessia…

Private sketches protect linear regression data privacy.

There is an especially strong need in modern large-scale data analysis to prioritize samples for manual inspection. For example, the inspection could target important mislabeled samples or key vulnerabilities exploitable by an adversarial attack. In order to solve the "needle in the haystack" problem of which samples t…

We show that a simple randomized sketch of the matrix multiplicative weight (MMW) update enjoys (in expectation) the same regret bounds as MMW, up to a small constant factor. Unlike MMW, where every step requires full matrix exponentiation, our steps require only a single product of the form $e^A b$, which the Lanczos …

Unified bounds for sketched bilinear forms in machine learning and statistics.

Matrix sketching is a recently developed data compression technique. An input matrix A is efficiently approximated with a smaller matrix B, so that B preserves most of the properties of A up to some guaranteed approximation ratio. In so doing numerical operations on big data sets become faster. Sketching algorithms gen…

A new method reduces the bias in estimating inverse covariance matrices from sketches.

We prove that two popular linear contextual bandit algorithms, OFUL and Thompson Sampling, can be made efficient using Frequent Directions, a deterministic online sketching technique. More precisely, we show that a sketch of size $m$ allows a $\mathcal{O}(md)$ update time for both algorithms, as opposed to $Ω(d^2)$ req…

Sketch-GNN reduces GNN training time and memory usage to sublinear scales.

The power of randomized algorithms in numerical methods have led to fast solutions which use the Singular Value Decomposition (SVD) as a core routine. However, given the large data size of modern and the modest runtime of SVD, most practical algorithms would require some form of approximation, such as sketching, when r…

A new method speeds up community detection in graphs.

Paper proposes an online covariance estimator for sketched Newton methods.

Tensor CANDECOMP/PARAFAC (CP) decomposition has wide applications in statistical learning of latent variable models and in data mining. In this paper, we propose fast and randomized tensor CP decomposition algorithms based on sketching. We build on the idea of count sketches, but introduce many novel ideas which are un…

The paper develops methods to estimate frequencies in large discrete data sets with improved coverage and robustness.

New algorithm converts data into sub-gaussian designs efficiently.

Sketching, a dimensionality reduction technique, has received much attention in the statistics community. In this paper, we study sketching in the context of Newton's method for solving finite-sum optimization problems in which the number of variables and data points are both large. We study two forms of sketching that…

Sketch-based approach detects community events in evolving networks.

New method reduces density estimation variance for multivariate data.

Sketch-BERT learns vector sketches using BERT-like self-supervised learning.

New method combines Monte Carlo and tensor networks for solving complex equations.

Scalable algorithms to solve optimization and regression tasks even approximately, are needed to work with large datasets. In this paper we study efficient techniques from matrix sketching to solve a variety of convex constrained regression problems. We adopt "Iterative Hessian Sketching" (IHS) and show that the fast C…

We consider the problem of finding anomalies in high-dimensional data using popular PCA based anomaly scores. The naive algorithms for computing these scores explicitly compute the PCA of the covariance matrix which uses space quadratic in the dimensionality of the data. We give the first streaming algorithms that use …

New algorithm speeds up polynomial kernel approximations.

Randomized algorithm solves vector-valued regression problems with low-rank operators.

Deep neural networks are powerful learning models that achieve state-of-the-art performance on many computer vision, speech, and language processing tasks. In this paper, we study a fundamental question that arises when designing deep network architectures: Given a target network architecture can we design a smaller ne…

A new method improves convergence in low-rank approximation.

We present a mechanism to compute a sketch (succinct summary) of how a complex modular deep network processes its inputs. The sketch summarizes essential information about the inputs and outputs of the network and can be used to quickly identify key components and summary statistics of the inputs. Furthermore, the sket…

Paper improves tensor approximation for streaming data.

Many popular first-order optimization methods (e.g., Momentum, AdaGrad, Adam) accelerate the convergence rate of deep learning models. However, these algorithms require auxiliary parameters, which cost additional memory proportional to the number of parameters in the model. The problem is becoming more severe as deep l…

Localized sketching improves matrix multiplication and ridge regression complexity.

We introduce a new sub-linear space sketch---the Weight-Median Sketch---for learning compressed linear classifiers over data streams while supporting the efficient recovery of large-magnitude weights in the model. This enables memory-limited execution of several statistical analyses over streams, including online featu…

The paper sharpens the analysis of sketch-and-project methods using randomized singular value decomposition.

Paper develops a method for estimating PFLM with minimized rates in high dimensions.

New guarantees for asymmetric sketching in compressive learning.

New analysis proves sketching operators' RIP guarantees for mixture models without importance sampling.

A fast sketching algorithm solves regularized least squares problems efficiently.

Corrects bias in random sampling matrices for improved ML methods.

A new method for estimating large-scale linear models with improved precision.

A fast algorithm for generalized matrix regression improves machine learning performance.

Sketching is a randomized dimensionality-reduction method that aims to preserve relevant information in large-scale datasets. Count sketch is a simple popular sketch which uses a randomized hash function to achieve compression. In this paper, we propose a novel extension known as Higher-order Count Sketch (HCS). While …