arXiv research

A new method improves convergence in low-rank approximation.

The paper sharpens the analysis of sketch-and-project methods using randomized singular value decomposition.

A new method for estimating large-scale linear models with improved precision.

Sketching accelerates structured prediction methods for large datasets.

Paper develops a bootstrap method for estimating sketched SVD errors.

This paper speeds up kernel methods using sparsified Gaussian sketches.

Private sketches protect linear regression data privacy.

New method reduces linear regret in high-dimensional bandit problems.

We introduce a new sub-linear space sketch---the Weight-Median Sketch---for learning compressed linear classifiers over data streams while supporting the efficient recovery of large-magnitude weights in the model. This enables memory-limited execution of several statistical analyses over streams, including online featu…

Iterative Hessian sketch (IHS) is an effective sketching method for modeling large-scale data. It was originally proposed by Pilanci and Wainwright (2016; JMLR) based on randomized sketching matrices. However, it is computationally intensive due to the iterative sketch process. In this paper, we analyze the IHS algorit…

Sketching reduces data size for accurate spectral estimation.

Paper proposes an online covariance estimator for sketched Newton methods.

Linear regression is a classic method of data analysis. In recent years, sketching -- a method of dimension reduction using random sampling, random projections, or both -- has gained popularity as an effective computational approximation when the number of observations greatly exceeds the number of variables. In this p…

We present sketch-rnn, a recurrent neural network (RNN) able to construct stroke-based drawings of common objects. The model is trained on thousands of crude human-drawn images representing hundreds of classes. We outline a framework for conditional and unconditional sketch generation, and describe new robust training …

Sketch-BERT learns vector sketches using BERT-like self-supervised learning.

New method for efficient sketching of gradients and Hessians.

Flexible method for estimating frequencies in large datasets using sketching.

We accelerate the power method for strong low-rank approximation using fast sketching.

Sketch-GNN reduces GNN training time and memory usage to sublinear scales.

Sketching is a randomized dimensionality-reduction method that aims to preserve relevant information in large-scale datasets. Count sketch is a simple popular sketch which uses a randomized hash function to achieve compression. In this paper, we propose a novel extension known as Higher-order Count Sketch (HCS). While …

Large-scale distributed training of neural networks is often limited by network bandwidth, wherein the communication time overwhelms the local computation time. Motivated by the success of sketching methods in sub-linear/streaming algorithms, we introduce Sketched SGD, an algorithm for carrying out distributed SGD by c…

FROSH and DFROSH speed up online sketching hashing for big data.

Sketching techniques have become popular for scaling up machine learning algorithms by reducing the sample size or dimensionality of massive data sets, while still maintaining the statistical power of big data. In this paper, we study sketching from an optimization point of view: we first show that the iterative Hessia…

A parallel optimization method for convex functions using Hessian sketching and debiasing.

Paper presents a new trie for integer sketches to improve similarity searches.

One-pass private sketch supports various machine learning tasks.

Tensor CANDECOMP/PARAFAC (CP) decomposition has wide applications in statistical learning of latent variable models and in data mining. In this paper, we propose fast and randomized tensor CP decomposition algorithms based on sketching. We build on the idea of count sketches, but introduce many novel ideas which are un…

A new method reduces the complexity of tensor products from cubic to quadratic, improving both speed and accuracy.

Develops precise expressions for random projections for better machine learning tasks.

The paper develops methods to estimate frequencies in large discrete data sets with improved coverage and robustness.

Newton-LESS sparsifies Gaussian sketching for faster optimization.

We address the statistical and optimization impacts of the classical sketch and Hessian sketch used to approximately solve the Matrix Ridge Regression (MRR) problem. Prior research has quantified the effects of classical sketch on the strictly simpler least squares regression (LSR) problem. We establish that classical …

We present a mechanism to compute a sketch (succinct summary) of how a complex modular deep network processes its inputs. The sketch summarizes essential information about the inputs and outputs of the network and can be used to quickly identify key components and summary statistics of the inputs. Furthermore, the sket…

New algorithm converts data into sub-gaussian designs efficiently.

We present a probabilistic model for Sketch-Based Image Retrieval (SBIR) where, at retrieval time, we are given sketches from novel classes, that were not present at training time. Existing SBIR methods, most of which rely on learning class-wise correspondences between sketches and images, typically work well only for …

We show that a simple randomized sketch of the matrix multiplicative weight (MMW) update enjoys (in expectation) the same regret bounds as MMW, up to a small constant factor. Unlike MMW, where every step requires full matrix exponentiation, our steps require only a single product of the form $e^A b$, which the Lanczos …

Localized sketching improves matrix multiplication and ridge regression complexity.

Improved privacy-preserving linear regression via iterative Hessian mixing.

Sketching, a dimensionality reduction technique, has received much attention in the statistics community. In this paper, we study sketching in the context of Newton's method for solving finite-sum optimization problems in which the number of variables and data points are both large. We study two forms of sketching that…

A new method reduces the bias in estimating inverse covariance matrices from sketches.

We derive optimal statistical and computational complexity bounds for exp-concave stochastic minimization in terms of the effective dimension. For common eigendecay patterns of the population covariance matrix, this quantity is significantly smaller than the ambient dimension. Our results reveal interesting connections…

New guarantees for asymmetric sketching in compressive learning.

New technique debiases distributed optimization, improving convergence rate.

The immense amount of daily generated and communicated data presents unique challenges in their processing. Clustering, the grouping of data without the presence of ground-truth labels, is an important tool for drawing inferences from data. Subspace clustering (SC) is a relatively recent method that is able to successf…

We consider statistical as well as algorithmic aspects of solving large-scale least-squares (LS) problems using randomized sketching algorithms. For a LS problem with input data $(X, Y) \in \mathbb{R}^{n \times p} \times \mathbb{R}^n$, sketching algorithms use a sketching matrix, $S\in\mathbb{R}^{r \times n}$ with $r \…

Improved estimator for least squares using random projections achieves smaller error.

A new method reduces the cost of solving large-scale linear models.

DiffSketch combines privacy and communication efficiency in distributed learning.