arXiv research

Illustrates interleaved learning with Kalman Filter for linear least squares.

CD converges linearly for MCP/SCAD penalized least squares.

The paper improves Kaczmarz algorithm with momentum for linear least squares.

Sparse linear regression, which entails finding a sparse solution to an underdetermined system of linear equations, can formally be expressed as an $l_0$-constrained least-squares problem. The Orthogonal Least-Squares (OLS) algorithm sequentially selects the features (i.e., columns of the coefficient matrix) to greedil…

A new method for estimating large-scale linear models with improved precision.

Least squares estimator fails to achieve optimal risk in bounded distributions, but non-linear predictors can.

Study identifies and validates a method for system identification of Markov jump linear systems.

This book introduces linear models and their theories rigorously.

Unified analysis of reweighted least-squares algorithms for linear models.

A new algorithm solves nonnegative least squares faster with nonnegative data.

A set of introductory notes on the subject of data classification using a linear classifier and least-squares cost function, and the negative effect of the presence of outliers on the decision boundary of the linear discriminant. We also show how a simple scaling could make the outlier less significant, thereby obtaini…

This paper reviews and compares supervised linear dimension-reduction techniques.

Optimal multiscale learning of linear operators

Paper proposes a pre-conditioning technique to speed up gradient-descent convergence in distributed linear least-squares problems.

ESNs trained with Tikhonov least squares approximate ergodic dynamical systems in L2(μ) norm.

Paper introduces $\ell$-DER for regression tasks using morphological operators and convex-concave procedure.

Lecture notes on advanced linear regression methods.

Ordinary least squares (OLS) is the default method for fitting linear models, but is not applicable for problems with dimensionality larger than the sample size. For these problems, we advocate the use of a generalized version of OLS motivated by ridge regression, and propose two novel three-step algorithms involving l…

Algorithm solves robust linear regression with block Lewis weights.

This paper optimizes sampling for least-squares approximation.

Estimates time-varying parameters from two OLS estimates.

The family of temporal difference (TD) methods span a spectrum from computationally frugal linear methods like TD(λ) to data efficient least squares methods. Least square methods make the best use of available data directly computing the TD solution and thus do not require tuning a typically highly sensitive learning r…

Optimal hashing embeddings reduce linear least squares solving time.

The paper explores how overfitting can lead to better predictions in high-dimensional data.

Logarithmic regret achieved in continuous-time linear-quadratic reinforcement learning.

StarNet trains deep models without gradients using linear equations.

The least-squares support vector machine is a frequently used kernel method for non-linear regression and classification tasks. Here we discuss several approximation algorithms for the least-squares support vector machine classifier. The proposed methods are based on randomized block kernel matrices, and we show that t…

OLS is a special case of Transformer, revealing its linear nature.

The OLS estimator optimally identifies stable linear systems with a finite number of samples.

We provide a pointwise confidence bound for non-linear least-squares with fixed design.

Efficient RL for linear MDPs with unknown transitions.

Least-squares models such as linear regression and Linear Discriminant Analysis (LDA) are amongst the most popular statistical learning techniques. However, since their computation time increases cubically with the number of features, they are inefficient in high-dimensional neuroimaging datasets. Fortunately, for k-fo…

This study examines the relationship between PLS and OLS regression using eigenvalue distributions.

DPLS improves asset pricing by capturing non-linear risk factor structures.

Subsampling methods have been recently proposed to speed up least squares estimation in large scale settings. However, these algorithms are typically not robust to outliers or corruptions in the observed covariates. The concept of influence that was developed for regression diagnostics can be used to detect such corrup…

We propose a novel algorithm for greedy forward feature selection for regularized least-squares (RLS) regression and classification, also known as the least-squares support vector machine or ridge regression. The algorithm, which we call greedy RLS, starts from the empty feature set, and on each iteration adds the feat…

Consider linear regression where the examples are generated by an unknown distribution on $R^d\times R$. Without any assumptions on the noise, the linear least squares solution for any i.i.d. sample will typically be biased w.r.t. the least squares optimum over the entire distribution. However, we show that if an i.i.d…

We analyze a simple prefiltered variation of the least squares estimator for the problem of estimation with biased, semi-parametric noise, an error model studied more broadly in causal statistics and active learning. We prove an oracle inequality which demonstrates that this procedure provably mitigates the variance in…

We describe stochastic Newton and stochastic quasi-Newton approaches to efficiently solve large linear least-squares problems where the very large data sets present a significant computational burden (e.g., the size may exceed computer memory or data are collected in real-time). In our proposed framework, stochasticity…

Efficiently estimates private least squares with linear error growth.

What are the prices of random variables? In this paper, we define the least-squares prices of coin-flipping games, which are proved to be minimal, positive linear, and arbitrage-free. These prices depend both on a set of games that are available for investing simultaneously and on a risk-free interest rate. In addition…

Optimal noise excitation for linear system identification reduces sample complexity.

We prove the statistical consistency of kernel Partial Least Squares Regression applied to a bounded regression learning problem on a reproducing kernel Hilbert space. Partial Least Squares stands out of well-known classical approaches as e.g. Ridge Regression or Principal Components Regression, as it is not defined as…

Proposes a partitioned least squares model for feature grouping.

It is shown that the the popular least squares method of option pricing converges even under very general assumptions. This substantially increases the freedom of creating different implementations of the method, with varying levels of computational complexity and flexible approach to regression. It is also argued that…

Bagging stabilizes linear interpolators, improving their generalization performance.

We prove a new and general concentration inequality for the excess risk in least-squares regression with random design and heteroscedastic noise. No specific structure is required on the model, except the existence of a suitable function that controls the local suprema of the empirical process. So far, only the case of…

We learn linear models from nonlinear systems using multiple trajectories and regularization.