arXiv research

The paper examines prediction and estimation risks of ridgeless least squares under general error assumptions.

Lecture notes on advanced linear regression methods.

Downsampling can improve generalization in ridgeless linear regression, especially with optimal sketching size.

Interpolators -- estimators that achieve zero training error -- have attracted growing attention in machine learning, mainly because state-of-the art neural networks appear to be models of this type. In this paper, we study minimum $\ell_2$ norm ("ridgeless") interpolation in high-dimensional least squares regression. …

Lower bound proves ridgeless regression performs poorly near interpolation threshold.

Study shows that ridgeless Gaussian kernel regression overfits even with varying bandwidth or dimensionality.

Ridgeless ReLU networks interpolate datasets and extrapolate based on curvature signs.

Kernel ridgeless regression with random features shows good generalization without explicit regularization.

The paper analyzes bagging in overparameterized learning, deriving risk properties and optimal subsample sizes.

Paper explores how DPP sampling can implicitly regularize kernel regression.

Enhanced kernel ridgeless regression improves performance with LAB RBF kernels.

Linear models can be poisoned by shifting a fraction of one class's data, revealing scaling laws and weight alignment.

Study ridge ensembles in proportional feature-to-sample size regime, proving risk equivalence and GCV consistency.

Data splitting enhances model performance in overparametrized ridgeless regression.

New bounds for KRR condition number reveal overfitting phenomena.

New method stabilizes machine learning for physics-informed inverse problems.

CD converges linearly for MCP/SCAD penalized least squares.

Interpolating models can have heavy-tailed risk, leading to rare but severe errors.

New method corrects least-squares temporal difference for better lambda-return estimation.

Illustrates interleaved learning with Kalman Filter for linear least squares.

We study randomized sketching methods for approximately solving least-squares problem with a general convex constraint. The quality of a least-squares approximation can be assessed in different ways: either in terms of the value of the quadratic objective function (cost approximation), or in terms of some distance meas…

Cross validation residuals are well known for the ordinary least squares model. Here leave-M-out cross validation is extended to generalised least squares. The relationship between cross validation residuals and Cook's distance is demonstrated, in terms of an approximation to the difference in the generalised residual …

We compare the risk of ridge regression to a simple variant of ordinary least squares, in which one simply projects the data onto a finite dimensional subspace (as specified by a Principal Component Analysis) and then performs an ordinary (un-regularized) least squares regression in this subspace. This note shows that …

Consider the problem: given the data pair $(\mathbf{x}, \mathbf{y})$ drawn from a population with $f_*(x) = \mathbf{E}[\mathbf{y} | \mathbf{x} = x]$, specify a neural network model and run gradient flow on the weights over time until reaching any stationarity. How does $f_t$, the function computed by the neural network…

In the absence of explicit regularization, Kernel "Ridgeless" Regression with nonlinear kernels has the potential to fit the training data perfectly. It has been observed empirically, however, that such interpolated solutions can still generalize well on test data. We isolate a phenomenon of implicit regularization for…

The paper improves Kaczmarz algorithm with momentum for linear least squares.

New algorithm improves online binary classification with constant time complexity.

Reduced-rank method improves least-squares regression under output regularity.

Paper uses deep learning to solve PDEs without supervision.

Sharp analysis of knowledge distillation for high-dimensional regression.

Proposes a partitioned least squares model for feature grouping.

ESNs trained with Tikhonov least squares approximate ergodic dynamical systems in L2(μ) norm.

The kernel least mean squares (KLMS) algorithm is a computationally efficient nonlinear adaptive filtering method that "kernelizes" the celebrated (linear) least mean squares algorithm. We demonstrate that the least mean squares algorithm is closely related to the Kalman filtering, and thus, the KLMS can be interpreted…

A new algorithm solves nonnegative least squares faster with nonnegative data.

The paper identifies saddlepoints in unsupervised auto-encoding neural nets.

The paper proposes a least squares method for binary compressive sampling with low intrinsic dimension signals.

We propose a new forward-backward stochastic differential equation solver for high-dimensional derivatives pricing problems by combining deep learning solver with least square regression technique widely used in the least square Monte Carlo method for the valuation of American options. Our numerical experiments demonst…

We introduce a novel semi-supervised version of the least squares classifier. This implicitly constrained least squares (ICLS) classifier minimizes the squared loss on the labeled data among the set of parameters implied by all possible labelings of the unlabeled data. Unlike other discriminative semi-supervised method…

The paper analyzes the risk of a least squares estimator under a spike covariance model.

New method speeds up solving L0-regularized least-squares problems.

Least squares estimator fails to achieve optimal risk in bounded distributions, but non-linear predictors can.

Sparse linear regression, which entails finding a sparse solution to an underdetermined system of linear equations, can formally be expressed as an $l_0$-constrained least-squares problem. The Orthogonal Least-Squares (OLS) algorithm sequentially selects the features (i.e., columns of the coefficient matrix) to greedil…

We introduce the implicitly constrained least squares (ICLS) classifier, a novel semi-supervised version of the least squares classifier. This classifier minimizes the squared loss on the labeled data among the set of parameters implied by all possible labelings of the unlabeled data. Unlike other discriminative semi-s…

We prove the statistical consistency of kernel Partial Least Squares Regression applied to a bounded regression learning problem on a reproducing kernel Hilbert space. Partial Least Squares stands out of well-known classical approaches as e.g. Ridge Regression or Principal Components Regression, as it is not defined as…

Randomized matrix compression techniques, such as the Johnson-Lindenstrauss transform, have emerged as an effective and practical way for solving large-scale problems efficiently. With a focus on computational efficiency, however, forsaking solutions quality and accuracy becomes the trade-off. In this paper, we investi…

This book introduces linear models and their theories rigorously.

The ratio of two probability densities can be used for solving various machine learning tasks such as covariate shift adaptation (importance sampling), outlier detection (likelihood-ratio test), and feature selection (mutual information). Recently, several methods of directly estimating the density ratio have been deve…

The least-squares support vector machine is a frequently used kernel method for non-linear regression and classification tasks. Here we discuss several approximation algorithms for the least-squares support vector machine classifier. The proposed methods are based on randomized block kernel matrices, and we show that t…