arXiv research

Sharp bounds on ERM's minimal error in regression.

This work gives a simultaneous analysis of both the ordinary least squares estimator and the ridge regression estimator in the random design setting under mild assumptions on the covariate/response distributions. In particular, the analysis provides sharp results on the ``out-of-sample'' prediction error, as opposed to…

High-dimensional settings, where the data dimension ($d$) far exceeds the number of observations ($n$), are common in many statistical and machine learning applications. Methods based on $\ell_1$-relaxation, such as Lasso, are very popular for sparse recovery in these settings. Restricted Eigenvalue (RE) condition is a…

Study on discrepancy principle for learning algorithms in nonparametric regression.

Adaptive PCR improves panel data analysis with uniform guarantees.

Selecting input variables or design points for statistical models has been of great interest in adaptive design and active learning. Motivated by two scientific examples, this paper presents a strategy of selecting the design points for a regression model when the underlying regression function is discontinuous. The fi…

Least Squares Estimators are suboptimal for 5D convex functions.

We propose a new two stage algorithm LING for large scale regression problems. LING has the same risk as the well known Ridge Regression under the fixed design setting and can be computed much faster. Our experiments have shown that LING performs well in terms of both prediction accuracy and computational efficiency co…

Paper analyzes robustness of MDPDE under INH setups.

Efficiently recovers piecewise linear functions from noisy samples.

We propose LOCO, an algorithm for large-scale ridge regression which distributes the features across workers on a cluster. Important dependencies between variables are preserved using structured random projections which are cheap to compute and must only be communicated once. We show that LOCO obtains a solution which …

New algorithm identifies best arm in semiparametric bandits with near optimal efficiency.

We consider the problem of online linear regression on arbitrary deterministic sequences when the ambient dimension d can be much larger than the number of time rounds T. We introduce the notion of sparsity regret bound, which is a deterministic online counterpart of recent risk bounds derived in the stochastic setting…

Max-affine regression refers to a model where the unknown regression function is modeled as a maximum of $k$ unknown affine functions for a fixed $k \geq 1$. This generalizes linear regression and (real) phase retrieval, and is closely related to convex regression. Working within a non-asymptotic framework, we study th…

We identify and validate a model for PCR in high dimensions, improving prediction guarantees.

In this paper, we study a simple iterative method for finding the Dantzig selector, which was designed for linear regression problems. The method consists of two main stages. The first stage is to approximate the Dantzig selector through a fixed-point formulation of solutions to the Dantzig selector problem. The second…

Improved bounds for unbounded losses using transductive priors.

New bounds for KRR condition number reveal overfitting phenomena.

Given a finite family of functions, the goal of model selection aggregation is to construct a procedure that mimics the function from this family that is the closest to an unknown regression function. More precisely, we consider a general regression model with fixed design and measure the distance between functions by …

In many scientific disciplines structures in high-dimensional data have to be found, e.g., in stellar spectra, in genome data, or in face recognition tasks. In this work we present a novel approach to non-linear dimensionality reduction. It is based on fitting K-nearest neighbor regression to the unsupervised regressio…

The Lasso method is analyzed for high-dimensional regression with Gaussian designs, leading to new insights on its performance.

We propose computationally efficient encoders and decoders for lossy compression using a Sparse Regression Code. The codebook is defined by a design matrix and codewords are structured linear combinations of columns of this matrix. The proposed encoding algorithm sequentially chooses columns of the design matrix to suc…

Regularization is used to find a solution that both fits the data and is sufficiently smooth, and thereby is very effective for designing and refining learning algorithms. But the influence of its exponent remains poorly understood. In particular, it is unclear how the exponent of the reproducing kernel Hilbert space~(…

Purpose: Arterial Spin Labeling (ASL) is a quantitative, non-invasive alternative to perfusion imaging with contrast agents. Fixing values of certain model parameters in traditional ASL, which actually vary from region to region, may introduce bias in perfusion estimates. Adopting Magnetic Resonance Fingerprinting (MRF…

Lasso performs poorly with correlated covariates, but a rescaled approach fixes this.

We introduce single-set spectral sparsification as a deterministic sampling based feature selection technique for regularized least squares classification, which is the classification analogue to ridge regression. The method is unsupervised and gives worst-case guarantees of the generalization power of the classificati…

Paper tightens statistical aggregation results using local complexity.

Paper proposes a method for early stopping in regression using reproducing kernels.

Paper develops robust econometric methods for staggered adoption studies.

Spectrahedral regression fits convex functions via a non-convex optimization problem.

This work establishes always-valid risk bounds for online matrix completion.

The paper proposes an efficient nested simulation design using likelihood ratio method.

In experimental design, we are given a large collection of vectors, each with a hidden response value that we assume derives from an underlying linear model, and we wish to pick a small subset of the vectors such that querying the corresponding responses will lead to a good estimator of the model. A classical approach …

Paper proves convergence rates for Gaussian kernel ridge regression.

Study learns linear system dynamics from noisy bilinear data.

We propose a new method of estimation in high-dimensional linear regression model. It allows for very weak distributional assumptions including heteroscedasticity, and does not require the knowledge of the variance of random errors. The method is based on linear programming only, so that its numerical implementation is…

Enhances neural network regression performance by modeling weight and variance uncertainty.

The paper explores how to select data points for optimal learning performance.

Short proof shows how ridge regression works with random data.

A new method for feature selection robust to noise and design variability.

Bayesian model estimates treatment effects near cutoffs in regression discontinuity designs.

This paper carries out a large dimensional analysis of a variation of kernel ridge regression that we call \emph{centered kernel ridge regression} (CKRR), also known in the literature as kernel ridge regression with offset. This modified technique is obtained by accounting for the bias in the regression problem resulti…

Gradient descent converges to a small neighborhood of the true parameter in logistic regression with Gaussian design.

Proposes a robust estimator for RD designs.

Locally private online quantile regression method addresses privacy constraints.

New algorithms for model selection in linear bandits adapt to instance complexity.

We study the optimal design problems where the goal is to choose a set of linear measurements to obtain the most accurate estimate of an unknown vector in $d$ dimensions. We study the $A$-optimal design variant where the objective is to minimize the average variance of the error in the maximum likelihood estimate of th…

Exact and scalable algorithm for Gaussian process regression with Matérn correlations.