arXiv research

New method for efficient sketching of gradients and Hessians.

A parallel optimization method for convex functions using Hessian sketching and debiasing.

Sketching techniques have become popular for scaling up machine learning algorithms by reducing the sample size or dimensionality of massive data sets, while still maintaining the statistical power of big data. In this paper, we study sketching from an optimization point of view: we first show that the iterative Hessia…

Improved privacy-preserving linear regression via iterative Hessian mixing.

We address the statistical and optimization impacts of the classical sketch and Hessian sketch used to approximately solve the Matrix Ridge Regression (MRR) problem. Prior research has quantified the effects of classical sketch on the strictly simpler least squares regression (LSR) problem. We establish that classical …

Iterative Hessian sketch (IHS) is an effective sketching method for modeling large-scale data. It was originally proposed by Pilanci and Wainwright (2016; JMLR) based on randomized sketching matrices. However, it is computationally intensive due to the iterative sketch process. In this paper, we analyze the IHS algorit…

New technique debiases distributed optimization, improving convergence rate.

Sketching, a dimensionality reduction technique, has received much attention in the statistics community. In this paper, we study sketching in the context of Newton's method for solving finite-sum optimization problems in which the number of variables and data points are both large. We study two forms of sketching that…

Newton-LESS sparsifies Gaussian sketching for faster optimization.

Scalable algorithms to solve optimization and regression tasks even approximately, are needed to work with large datasets. In this paper we study efficient techniques from matrix sketching to solve a variety of convex constrained regression problems. We adopt "Iterative Hessian Sketching" (IHS) and show that the fast C…

Develops unbiased averaging methods for second order optimization in distributed systems.

We study randomized sketching methods for approximately solving least-squares problem with a general convex constraint. The quality of a least-squares approximation can be assessed in different ways: either in terms of the value of the quadratic objective function (cost approximation), or in terms of some distance meas…

Our work connects parameter magnitudes and Hessian eigenspaces in deep neural nets.

A new algorithm reduces the time and space complexity for multinomial logistic bandits.

New methods for sketching non-PSD matrices improve regression and optimization tasks.

Paper proposes an efficient online Newton method with Nesterov's acceleration for streaming data.

In this paper, we revisit the large-scale constrained linear regression problem and propose faster methods based on some recent developments in sketching and optimization. Our algorithms combine (accelerated) mini-batch SGD with a new method called two-step preconditioning to achieve an approximate solution with a time…

Proposes a new method for optimizing large-scale models using Nyström approximation of the Hessian.

New averaging technique speeds up Newton method convergence.

We propose a randomized second-order method for optimization known as the Newton Sketch: it is based on performing an approximate Newton step using a randomly projected or sub-sampled Hessian. For self-concordant functions, we prove that the algorithm has super-linear convergence with exponentially high probability, wi…

Paper tackles catastrophic forgetting in sequential learning.

Sketch-BERT learns vector sketches using BERT-like self-supervised learning.

SVRN accelerates Newton methods by reducing variance and improving performance.

A new method improves convergence in low-rank approximation.

We present a mechanism to compute a sketch (succinct summary) of how a complex modular deep network processes its inputs. The sketch summarizes essential information about the inputs and outputs of the network and can be used to quickly identify key components and summary statistics of the inputs. Furthermore, the sket…

Localized sketching improves matrix multiplication and ridge regression complexity.

We introduce a new sub-linear space sketch---the Weight-Median Sketch---for learning compressed linear classifiers over data streams while supporting the efficient recovery of large-magnitude weights in the model. This enables memory-limited execution of several statistical analyses over streams, including online featu…

The paper sharpens the analysis of sketch-and-project methods using randomized singular value decomposition.

New method reduces linear regret in high-dimensional bandit problems.

New guarantees for asymmetric sketching in compressive learning.

New analysis proves sketching operators' RIP guarantees for mixture models without importance sampling.

A fast sketching algorithm solves regularized least squares problems efficiently.

A new method for estimating large-scale linear models with improved precision.

Private sketches protect linear regression data privacy.

Sketching is a randomized dimensionality-reduction method that aims to preserve relevant information in large-scale datasets. Count sketch is a simple popular sketch which uses a randomized hash function to achieve compression. In this paper, we propose a novel extension known as Higher-order Count Sketch (HCS). While …

DiPS learns to optimize sketching policies for better recommendation quality.

Paper proposes Nyström sketches for better adaptive compressive learning.

Sketching is more fundamental to human cognition than speech. Deep Neural Networks (DNNs) have achieved the state-of-the-art in speech-related tasks but have not made significant development in generating stroke-based sketches a.k.a sketches in vector format. Though there are Variational Auto Encoders (VAEs) for genera…

Density sketches summarize data distributions for accurate sampling and estimation.

Improved sketching for logistic and $\ell_1$ regression with near-linear dimensions.

Unified bounds for sketched bilinear forms in machine learning and statistics.

We improve prediction risk estimation for large datasets using sketching and ridge regression.

Sketching accelerates structured prediction methods for large datasets.

Linear regression is a classic method of data analysis. In recent years, sketching -- a method of dimension reduction using random sampling, random projections, or both -- has gained popularity as an effective computational approximation when the number of observations greatly exceeds the number of variables. In this p…

Paper develops a bootstrap method for estimating sketched SVD errors.

We present sketch-rnn, a recurrent neural network (RNN) able to construct stroke-based drawings of common objects. The model is trained on thousands of crude human-drawn images representing hundreds of classes. We outline a framework for conditional and unconditional sketch generation, and describe new robust training …

Sketching reduces data size for accurate spectral estimation.

Downsampling can improve generalization in ridgeless linear regression, especially with optimal sketching size.