arXiv research

Reduced-rank method improves least-squares regression under output regularity.

Proposes a method for coarse graph alignment using sparse partial least squares.

New algorithms estimate Jacobian matrices for large-scale machine learning.

CD converges linearly for MCP/SCAD penalized least squares.

New algorithm extracts shared latent space for cortico-muscular interactions.

Unified analysis of reweighted least-squares algorithms for linear models.

Optimal multiscale learning of linear operators

R2T hybrid model improves robust regression for asymmetric noise.

The paper examines prediction and estimation risks of ridgeless least squares under general error assumptions.

Illustrates interleaved learning with Kalman Filter for linear least squares.

We study randomized sketching methods for approximately solving least-squares problem with a general convex constraint. The quality of a least-squares approximation can be assessed in different ways: either in terms of the value of the quadratic objective function (cost approximation), or in terms of some distance meas…

Cross validation residuals are well known for the ordinary least squares model. Here leave-M-out cross validation is extended to generalised least squares. The relationship between cross validation residuals and Cook's distance is demonstrated, in terms of an approximation to the difference in the generalised residual …

Study on learning sparse fixed-structure Gaussian Bayesian networks with near-optimal sample complexity.

We compare the risk of ridge regression to a simple variant of ordinary least squares, in which one simply projects the data onto a finite dimensional subspace (as specified by a Principal Component Analysis) and then performs an ordinary (un-regularized) least squares regression in this subspace. This note shows that …

We study risk of the minimum norm linear least squares estimator in when the number of parameters $d$ depends on $n$, and $\frac{d}{n} \rightarrow \infty$. We assume that data has an underlying low rank structure by restricting ourselves to spike covariance matrices, where a fixed finite number of eigenvalues grow with…

We prove a new and general concentration inequality for the excess risk in least-squares regression with random design and heteroscedastic noise. No specific structure is required on the model, except the existence of a suitable function that controls the local suprema of the empirical process. So far, only the case of…

We describe stochastic Newton and stochastic quasi-Newton approaches to efficiently solve large linear least-squares problems where the very large data sets present a significant computational burden (e.g., the size may exceed computer memory or data are collected in real-time). In our proposed framework, stochasticity…

DPLS improves asset pricing by capturing non-linear risk factor structures.

We consider least squares estimation in a general nonparametric regression model. The rate of convergence of the least squares estimator (LSE) for the unknown regression function is well studied when the errors are sub-Gaussian. We find upper bounds on the rates of convergence of the LSE when the errors have uniformly …

Estimates manifold dimension using local graph structure.

Lecture notes on advanced linear regression methods.

The paper improves Kaczmarz algorithm with momentum for linear least squares.

We study the total least squares (TLS) problem that generalizes least squares regression by allowing measurement errors in both dependent and independent variables. TLS is widely used in applied fields including computer vision, system identification and econometrics. The special case when all dependent and independent…

New algorithm improves online binary classification with constant time complexity.

Regularized least-squares (kernel-ridge / Gaussian process) regression is a fundamental algorithm of statistics and machine learning. Because generic algorithms for the exact solution have cubic complexity in the number of datapoints, large datasets require to resort to approximations. In this work, the computation of …

This paper studies statistical estimation in optional regression models.

Paper presents novel online MTL methods using WRLS and OSLSSVR.

Proposes a partitioned least squares model for feature grouping.

ESNs trained with Tikhonov least squares approximate ergodic dynamical systems in L2(μ) norm.

The kernel least mean squares (KLMS) algorithm is a computationally efficient nonlinear adaptive filtering method that "kernelizes" the celebrated (linear) least mean squares algorithm. We demonstrate that the least mean squares algorithm is closely related to the Kalman filtering, and thus, the KLMS can be interpreted…

A new algorithm solves nonnegative least squares faster with nonnegative data.

A general framework of least squares support vector machine with low rank kernels, referred to as LR-LSSVM, is introduced in this paper. The special structure of low rank kernels with a controlled model size brings sparsity as well as computational efficiency to the proposed model. Meanwhile, a two-step optimization al…

The paper identifies saddlepoints in unsupervised auto-encoding neural nets.

The paper proposes a least squares method for binary compressive sampling with low intrinsic dimension signals.

This paper studies an unsupervised deep learning-based numerical approach for solving partial differential equations (PDEs). The approach makes use of the deep neural network to approximate solutions of PDEs through the compositional construction and employs least-squares functionals as loss functions to determine para…

In this paper, the symplectic genus for any 2-dimensional class in a 4-manifold admitting a symplectic structure is introduced, and its relation with the minimal genus is studied. It is used to describe which classes in rational and irrational ruled manifolds are realized by connected symplectic surfaces. In particular…

GLSKF improves tensor completion by capturing both global and local variations.

We propose a new forward-backward stochastic differential equation solver for high-dimensional derivatives pricing problems by combining deep learning solver with least square regression technique widely used in the least square Monte Carlo method for the valuation of American options. Our numerical experiments demonst…

We introduce a novel semi-supervised version of the least squares classifier. This implicitly constrained least squares (ICLS) classifier minimizes the squared loss on the labeled data among the set of parameters implied by all possible labelings of the unlabeled data. Unlike other discriminative semi-supervised method…

New method speeds up solving L0-regularized least-squares problems.

Least squares estimator fails to achieve optimal risk in bounded distributions, but non-linear predictors can.

Principal component analysis (PCA) is often used to reduce the dimension of data by selecting a few orthonormal vectors that explain most of the variance structure of the data. L1 PCA uses the L1 norm to measure error, whereas the conventional PCA uses the L2 norm. For the L1 PCA problem minimizing the fitting error of…

Sparse linear regression, which entails finding a sparse solution to an underdetermined system of linear equations, can formally be expressed as an $l_0$-constrained least-squares problem. The Orthogonal Least-Squares (OLS) algorithm sequentially selects the features (i.e., columns of the coefficient matrix) to greedil…

We introduce the implicitly constrained least squares (ICLS) classifier, a novel semi-supervised version of the least squares classifier. This classifier minimizes the squared loss on the labeled data among the set of parameters implied by all possible labelings of the unlabeled data. Unlike other discriminative semi-s…

OLS predictions are shown to be similar to attention mechanisms in models.

This paper explores using SSIM for better image generation in generative models.

We prove the statistical consistency of kernel Partial Least Squares Regression applied to a bounded regression learning problem on a reproducing kernel Hilbert space. Partial Least Squares stands out of well-known classical approaches as e.g. Ridge Regression or Principal Components Regression, as it is not defined as…

Unified methodology for statistical inference in least squares and PCA via randomized sketching.