arXiv research

Illustrates interleaved learning with Kalman Filter for linear least squares.

CD converges linearly for MCP/SCAD penalized least squares.

The paper improves Kaczmarz algorithm with momentum for linear least squares.

This book introduces linear models and their theories rigorously.

Least squares estimator fails to achieve optimal risk in bounded distributions, but non-linear predictors can.

Sparse linear regression, which entails finding a sparse solution to an underdetermined system of linear equations, can formally be expressed as an $l_0$-constrained least-squares problem. The Orthogonal Least-Squares (OLS) algorithm sequentially selects the features (i.e., columns of the coefficient matrix) to greedil…

A new method for estimating large-scale linear models with improved precision.

A new algorithm solves nonnegative least squares faster with nonnegative data.

Study identifies and validates a method for system identification of Markov jump linear systems.

Unified analysis of reweighted least-squares algorithms for linear models.

ESNs trained with Tikhonov least squares approximate ergodic dynamical systems in L2(μ) norm.

Optimal multiscale learning of linear operators

This paper reviews and compares supervised linear dimension-reduction techniques.

A set of introductory notes on the subject of data classification using a linear classifier and least-squares cost function, and the negative effect of the presence of outliers on the decision boundary of the linear discriminant. We also show how a simple scaling could make the outlier less significant, thereby obtaini…

Paper proposes a pre-conditioning technique to speed up gradient-descent convergence in distributed linear least-squares problems.

Ordinary least squares (OLS) is the default method for fitting linear models, but is not applicable for problems with dimensionality larger than the sample size. For these problems, we advocate the use of a generalized version of OLS motivated by ridge regression, and propose two novel three-step algorithms involving l…

The kernel least mean squares (KLMS) algorithm is a computationally efficient nonlinear adaptive filtering method that "kernelizes" the celebrated (linear) least mean squares algorithm. We demonstrate that the least mean squares algorithm is closely related to the Kalman filtering, and thus, the KLMS can be interpreted…

This paper optimizes sampling for least-squares approximation.

Paper introduces $\ell$-DER for regression tasks using morphological operators and convex-concave procedure.

The family of temporal difference (TD) methods span a spectrum from computationally frugal linear methods like TD(λ) to data efficient least squares methods. Least square methods make the best use of available data directly computing the TD solution and thus do not require tuning a typically highly sensitive learning r…

The least-squares support vector machine is a frequently used kernel method for non-linear regression and classification tasks. Here we discuss several approximation algorithms for the least-squares support vector machine classifier. The proposed methods are based on randomized block kernel matrices, and we show that t…

Algorithm solves robust linear regression with block Lewis weights.

Lecture notes on advanced linear regression methods.

The OLS estimator optimally identifies stable linear systems with a finite number of samples.

The paper explores how overfitting can lead to better predictions in high-dimensional data.

Consider linear regression where the examples are generated by an unknown distribution on $R^d\times R$. Without any assumptions on the noise, the linear least squares solution for any i.i.d. sample will typically be biased w.r.t. the least squares optimum over the entire distribution. However, we show that if an i.i.d…

Optimal hashing embeddings reduce linear least squares solving time.

Least-squares models such as linear regression and Linear Discriminant Analysis (LDA) are amongst the most popular statistical learning techniques. However, since their computation time increases cubically with the number of features, they are inefficient in high-dimensional neuroimaging datasets. Fortunately, for k-fo…

Estimates time-varying parameters from two OLS estimates.

This study examines the relationship between PLS and OLS regression using eigenvalue distributions.

Efficiently estimates private least squares with linear error growth.

Proposes a partitioned least squares model for feature grouping.

Logarithmic regret achieved in continuous-time linear-quadratic reinforcement learning.

Subsampling methods have been recently proposed to speed up least squares estimation in large scale settings. However, these algorithms are typically not robust to outliers or corruptions in the observed covariates. The concept of influence that was developed for regression diagnostics can be used to detect such corrup…

We prove the statistical consistency of kernel Partial Least Squares Regression applied to a bounded regression learning problem on a reproducing kernel Hilbert space. Partial Least Squares stands out of well-known classical approaches as e.g. Ridge Regression or Principal Components Regression, as it is not defined as…

We provide a pointwise confidence bound for non-linear least-squares with fixed design.

We prove a new and general concentration inequality for the excess risk in least-squares regression with random design and heteroscedastic noise. No specific structure is required on the model, except the existence of a suitable function that controls the local suprema of the empirical process. So far, only the case of…

OLS is a special case of Transformer, revealing its linear nature.

We analyze a simple prefiltered variation of the least squares estimator for the problem of estimation with biased, semi-parametric noise, an error model studied more broadly in causal statistics and active learning. We prove an oracle inequality which demonstrates that this procedure provably mitigates the variance in…

Improved Least-Squares Monte Carlo with finite-difference ansatz.

We describe stochastic Newton and stochastic quasi-Newton approaches to efficiently solve large linear least-squares problems where the very large data sets present a significant computational burden (e.g., the size may exceed computer memory or data are collected in real-time). In our proposed framework, stochasticity…

We propose a novel algorithm for greedy forward feature selection for regularized least-squares (RLS) regression and classification, also known as the least-squares support vector machine or ridge regression. The algorithm, which we call greedy RLS, starts from the empty feature set, and on each iteration adds the feat…

StarNet trains deep models without gradients using linear equations.

What are the prices of random variables? In this paper, we define the least-squares prices of coin-flipping games, which are proved to be minimal, positive linear, and arbitrage-free. These prices depend both on a set of games that are available for investing simultaneously and on a risk-free interest rate. In addition…

Efficient RL for linear MDPs with unknown transitions.

DPLS improves asset pricing by capturing non-linear risk factor structures.

It is shown that the the popular least squares method of option pricing converges even under very general assumptions. This substantially increases the freedom of creating different implementations of the method, with varying levels of computational complexity and flexible approach to regression. It is also argued that…

The discovery of non-linear causal relationship under additive non-Gaussian noise models has attracted considerable attention recently because of their high flexibility. In this paper, we propose a novel causal inference algorithm called least-squares independence regression (LSIR). LSIR learns the additive noise model…