arXiv research

Optimal sketching bounds for sparse linear regression under various loss functions are established.

A fast sketching algorithm solves regularized least squares problems efficiently.

The paper sharpens the analysis of sketch-and-project methods using randomized singular value decomposition.

We study how well one can recover sparse principal components of a data matrix using a sketch formed from a few of its elements. We show that for a wide class of optimization problems, if the sketch is close (in the spectral norm) to the original data matrix, then one can recover a near optimal solution to the optimiza…

FetchSGD reduces communication in federated learning with sketching.

Scalable algorithms to solve optimization and regression tasks even approximately, are needed to work with large datasets. In this paper we study efficient techniques from matrix sketching to solve a variety of convex constrained regression problems. We adopt "Iterative Hessian Sketching" (IHS) and show that the fast C…

A new method reduces the bias in estimating inverse covariance matrices from sketches.

Optimal hashing embeddings reduce linear least squares solving time.

This paper speeds up kernel methods using sparsified Gaussian sketches.

In this paper, we propose a general framework for sparse and low-rank tensor estimation from cubic sketchings. A two-stage non-convex implementation is developed based on sparse tensor decomposition and thresholded gradient descent, which ensures exact recovery in the noiseless case and stable recovery in the noisy cas…

Tensor CANDECOMP/PARAFAC (CP) decomposition has wide applications in statistical learning of latent variable models and in data mining. In this paper, we propose fast and randomized tensor CP decomposition algorithms based on sketching. We build on the idea of count sketches, but introduce many novel ideas which are un…

Paper advances sparse regularisation theory for measures with new kernel insights.

We consider the problem of selecting non-zero entries of a matrix $A$ in order to produce a sparse sketch of it, $B$, that minimizes $\|A-B\|_2$. For large $m \times n$ matrices, such that $n \gg m$ (for example, representing $n$ observations over $m$ attributes) we give sampling distributions that exhibit four importa…

Improved COD algorithm reduces streaming AMM errors and uses less space.

New sketches for weighted $\ell_p$ sampling without replacement improve accuracy and efficiency.

New algorithm optimizes positions of CountSketch non-zero entries for better data compression.

We learn sparse precision matrices from compressed data sketches.

Paper develops a decoder for sparse codes without encoder matrix, achieving optimal recovery.

New algorithm reduces matrix multiplication time for sparse matrices.

This paper explores and analyzes two randomized designs for robust Principal Component Analysis (PCA) employing low-dimensional data sketching. In one design, a data sketch is constructed using random column sampling followed by low dimensional embedding, while in the other, sketching is based on random column and row …

Unified methodology for statistical inference in least squares and PCA via randomized sketching.

Corrects bias in random sampling matrices for improved ML methods.

Sketch-BERT learns vector sketches using BERT-like self-supervised learning.

Learning parameters from voluminous data can be prohibitive in terms of memory and computational requirements. We propose a "compressive learning" framework where we estimate model parameters from a sketch of the training data. This sketch is a collection of generalized moments of the underlying probability distributio…

A new method improves convergence in low-rank approximation.

We address the statistical and optimization impacts of the classical sketch and Hessian sketch used to approximately solve the Matrix Ridge Regression (MRR) problem. Prior research has quantified the effects of classical sketch on the strictly simpler least squares regression (LSR) problem. We establish that classical …

We present a mechanism to compute a sketch (succinct summary) of how a complex modular deep network processes its inputs. The sketch summarizes essential information about the inputs and outputs of the network and can be used to quickly identify key components and summary statistics of the inputs. Furthermore, the sket…

Localized sketching improves matrix multiplication and ridge regression complexity.

We introduce a new sub-linear space sketch---the Weight-Median Sketch---for learning compressed linear classifiers over data streams while supporting the efficient recovery of large-magnitude weights in the model. This enables memory-limited execution of several statistical analyses over streams, including online featu…

New method reduces linear regret in high-dimensional bandit problems.

New guarantees for asymmetric sketching in compressive learning.

New analysis proves sketching operators' RIP guarantees for mixture models without importance sampling.

A new method for estimating large-scale linear models with improved precision.

Private sketches protect linear regression data privacy.

Sketching is a randomized dimensionality-reduction method that aims to preserve relevant information in large-scale datasets. Count sketch is a simple popular sketch which uses a randomized hash function to achieve compression. In this paper, we propose a novel extension known as Higher-order Count Sketch (HCS). While …

DiPS learns to optimize sketching policies for better recommendation quality.

Paper proposes Nyström sketches for better adaptive compressive learning.

Sketching is more fundamental to human cognition than speech. Deep Neural Networks (DNNs) have achieved the state-of-the-art in speech-related tasks but have not made significant development in generating stroke-based sketches a.k.a sketches in vector format. Though there are Variational Auto Encoders (VAEs) for genera…

Density sketches summarize data distributions for accurate sampling and estimation.

Improved sketching for logistic and $\ell_1$ regression with near-linear dimensions.

Sparse matrices are favorable objects in machine learning and optimization. When such matrices are used, in place of dense ones, the overall complexity requirements in optimization can be significantly reduced in practice, both in terms of space and run-time. Prompted by this observation, we study a convex optimization…

Unified bounds for sketched bilinear forms in machine learning and statistics.

We improve prediction risk estimation for large datasets using sketching and ridge regression.

We introduce a "learning-based" algorithm for the low-rank decomposition problem: given an $n \times d$ matrix $A$, and a parameter $k$, compute a rank-$k$ matrix $A'$ that minimizes the approximation loss $\|A-A'\|_F$. The algorithm uses a training set of input matrices in order to optimize its performance. Specifical…

Sketching accelerates structured prediction methods for large datasets.

Linear regression is a classic method of data analysis. In recent years, sketching -- a method of dimension reduction using random sampling, random projections, or both -- has gained popularity as an effective computational approximation when the number of observations greatly exceeds the number of variables. In this p…

Paper develops a bootstrap method for estimating sketched SVD errors.

Sketching techniques have become popular for scaling up machine learning algorithms by reducing the sample size or dimensionality of massive data sets, while still maintaining the statistical power of big data. In this paper, we study sketching from an optimization point of view: we first show that the iterative Hessia…