We compare the risk of ridge regression to a simple variant of ordinary least squares, in which one simply projects the data onto a finite dimensional subspace (as specified by a Principal Component Analysis) and then performs an ordinary (un-regularized) least squares regression in this subspace. This note shows that …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “least squares regression”
Reduced-rank method improves least-squares regression under output regularity.
problem Least-squares regression with infinite dimensional outputs.
method Reduced-rank method for solving least-squares problems with output regularity assumptions.
result Learning bounds and improved statistical performance compared to full-rank method.
We find a convex model for traditional nonlinear regression under L2 loss.
problem Nonlinear regression under L2 loss with non-convex optimization.
method Showed a convex nonlinear regression model for least squares problem.
result Existence of a convex model simplifies training complex systems.
We prove the statistical consistency of kernel Partial Least Squares Regression applied to a bounded regression learning problem on a reproducing kernel Hilbert space. Partial Least Squares stands out of well-known classical approaches as e.g. Ridge Regression or Principal Components Regression, as it is not defined as…
Improved robust regression for heavy-tailed and contaminated data.
problem Linear regression with heavy-tailed and adversarially contaminated covariates and responses.
method Applying a filtering algorithm to covariates and then using Huber regression, least trimmed squares, or least absolute deviation estimators on the remaining data.
result Near-optimal error rates achieved for the Huber regression estimator.
We prove rates of convergence in the statistical sense for kernel-based least squares regression using a conjugate gradient algorithm, where regularization against overfitting is obtained by early stopping. This method is directly related to Kernel Partial Least Squares, a regression method that combines supervised dim…
R2T hybrid model improves robust regression for asymmetric noise.
problem Least-squares regression fails with asymmetric structured noise.
method Transformer encoder, compression NN, fixed symbolic equation.
result Median regression MSE of 6e-6 to 3.5e-5 on synthetic data.
This study examines the relationship between PLS and OLS regression using eigenvalue distributions.
problem Analyzing the difference between PLS and OLS regression in terms of eigenvalue distributions.
method Examined the distance between PLS and OLS regression coefficients using the Mahalanobis distance and eigenvalue distributions of the regressor covariance matrix.
result Provided a bound on the distance between PLS and OLS regression coefficients that depends only on the eigenvalue distribution of the regressor covariance matrix.
Sparse linear regression, which entails finding a sparse solution to an underdetermined system of linear equations, can formally be expressed as an -constrained least-squares problem. The Orthogonal Least-Squares (OLS) algorithm sequentially selects the features (i.e., columns of the coefficient matrix) to greedil…
This paper reviews SDR methods for multivariate response regression.
problem Handling sufficient dimension reduction for multivariate response regression.
method Characterizes SDR estimators as inverse or forward regression methods.
result Pooled marginal, projective resampling, distance-based, ordinary least squares, partial least squares, and semiparametric SDR estimators are discussed.
The paper examines prediction and estimation risks of ridgeless least squares under general error assumptions.
problem Prediction and estimation risks of ridgeless least squares under realistic error structures.
method Analysis of prediction and estimation risks under general regression error assumptions, including clustered or serial dependence.
result The benefits of overparameterization extend to time series, panel, and grouped data.
ESNs trained with Tikhonov least squares approximate ergodic dynamical systems in L2(μ) norm.
problem Approximating ergodic dynamical systems using ESNs.
method Tikhonov least squares regression on ESNs trained on observations from an ergodic dynamical system.
result ESNs trained with Tikhonov least squares approximate the target function in the L2(μ) norm.
Local Control Regression: Improving the Least Squares Monte Carlo Method for Portfolio Optimizationq-fin.PM
The least squares Monte Carlo algorithm has become popular for solving portfolio optimization problems. A simple approach is to approximate the value functions on a discrete grid of portfolio weights, then use control regression to generalize the discrete estimates. However, the classical global control regression can …
Lecture notes on advanced linear regression methods.
problem Understanding the properties of linear regression estimators in high dimensions.
method Proposition-proof exploration of least squares, ridgeless, ridge, and lasso estimators.
result Detailed analysis of the existence, uniqueness, relations, computation, and non-asymptotic properties of these estimators.
Algorithm solves robust linear regression with block Lewis weights.
problem Group distributionally robust least squares problem.
method Algorithm based on geometric construction and block Lewis weights, using accelerated proximal methods.
result Improves over known methods for moderate accuracy regimes and matches state-of-the-art guarantees.
Efficiently estimates private least squares with linear error growth.
problem Private estimation of ordinary least squares with bounded residuals and leverage.
method Scaled noise added to a stable nonprivate estimator of the regression vector.
result Near-optimal accuracy guarantee with linear error growth in dimension.
Many problems in financial engineering involve the estimation of unknown conditional expectations across a time interval. Often Least Squares Monte Carlo techniques are used for the estimation. One method that can be combined with Least Squares Monte Carlo is the "Regress-Later" method. Unlike conventional methods wher…
This paper explores alternative regression techniques in pricing American put options and compares to the least-squares method (LSM) in Monte Carlo implemented by Longstaff-Schwartz, 2001 which uses least squares to estimate the conditional expected payoff to the option holder from continuation. The pricing is done und…
Least squares kernel based methods have been widely used in regression problems due to the simple implementation and good generalization performance. Among them, least squares support vector regression (LS-SVR) and extreme learning machine (ELM) are popular techniques. However, the noise sensitivity is a major bottlene…
In this paper, we consider the nonparametric least square regression in a Reproducing Kernel Hilbert Space (RKHS). We propose a new randomized algorithm that has optimal generalization error bounds with respect to the square loss, closing a long-standing gap between upper and lower bounds. Moreover, we show that our al…
Least Squares Estimators are suboptimal for 5D convex functions.
problem Suboptimality of Least Squares Estimators in estimating multidimensional convex functions.
method Analysis of natural subclasses of convex functions in random and fixed design settings.
result Risk of LSE is while minimax risk is for .
Optimal multiscale learning of linear operators
problem Statistical and computational limits of learning bounded linear operators between Sobolev spaces
method Reformulate as an infinite-dimensional matrix regression problem with heterogeneous multiscale structure
result Establish minimax rates and construct a finite-resolution blockwise least-squares estimator attaining these rates
Study on consistency of ML methods for moving objects in non-stationary environments.
problem Consistency of machine learning methods for moving objects in non-stationary environments.
method Least squares, ridge regression, and -penalized least squares methods under non-stationary spatial-temporal sampling.
result Consistency and asymptotic normality of the estimates under weak conditions.
We prove statistical rates of convergence for kernel-based least squares regression from i.i.d. data using a conjugate gradient algorithm, where regularization against overfitting is obtained by early stopping. This method is related to Kernel Partial Least Squares, a regression method that combines supervised dimensio…
Paper introduces -DER for regression tasks using morphological operators and convex-concave procedure.
problem Developing a universal approximator for regression tasks.
method Introduces -DER model, trains it using a convex-concave procedure (CCP) to minimize least-squares.
result Outperforms other hybrid morphological models and state-of-the-art approaches.
Improved regression analysis using Padé approximants with new residuals and regularization.
problem Improving regression analysis with Padé approximants for accuracy and avoiding overfitting.
method New residuals in least squares method, system of linear equations for rational functions, Tikhonov regularization.
result Demonstrated efficiency in practical cases from physics and reliability theory.
It is shown that the the popular least squares method of option pricing converges even under very general assumptions. This substantially increases the freedom of creating different implementations of the method, with varying levels of computational complexity and flexible approach to regression. It is also argued that…
Least squares regression shows unexpected double descent in under-parameterized models.
problem Understanding the generalization of under-parameterized models in regression.
method Analyzing the spectrum and eigenvectors of the sample covariance matrix.
result Least squares regression can exhibit a peak in generalization in the under-parameterized regime, contrary to previous explanations.
Proposes a new regression method using -norms for non-Gaussian noise.
problem Non-Gaussian noise in residuals affects the performance of local least squares regression.
method Introduces local polynomial -norm regression, replacing weighted least squares with weighted -norm estimation.
result Demonstrates superior performance over local least squares in one-dimensional data and higher dimensions.
Improved SGD for non-strongly-convex regression with faster convergence.
problem Non-strongly-convex least squares regression problems.
method Modified accelerated gradient descent.
result Achieves optimal prediction error rates of and forgets initial conditions faster to .
Error dynamics of mini-batch gradient descent with random reshuffling for least squares regressionstat.ML
Study shows how mini-batch GD with random reshuffling affects least squares regression dynamics.
problem Analyzing the error dynamics of mini-batch GD with random reshuffling for least squares regression.
method Represented training and generalization errors through a sample cross-covariance matrix Z, compared with sample covariance matrix of original features X, and used linear scaling rule for analysis.
result Mini-batch GD with random reshuffling exhibits subtle step-size dependence not detectable by gradient flow analysis, converging to a limit dependent on the step size.
New method for unbiased regression reduces excess risk.
problem Least squares regression with optimal solution and Hessian matrix.
method Averaged stochastic gradient descent with time-average estimator.
result Unbiased estimator with O(1/k) expected excess risk.
The paper provides bounds for regression schemes using nonstationary training samples.
problem Developing confidence intervals for nonparametric regression with nonstationary data.
method The approach involves Rademacher and Vapnik-Chervonenkis theories to analyze the cost and optimality of regression schemes.
result The paper establishes nonasymptotic bounds for regression schemes and optimality in -distance.
The derivation of statistical properties for Partial Least Squares regression can be a challenging task. The reason is that the construction of latent components from the predictor variables also depends on the response variable. While this typically leads to good performance and interpretable models in practice, it ma…
Consider linear regression where the examples are generated by an unknown distribution on . Without any assumptions on the noise, the linear least squares solution for any i.i.d. sample will typically be biased w.r.t. the least squares optimum over the entire distribution. However, we show that if an i.i.d…
The least-squares support vector machine is a frequently used kernel method for non-linear regression and classification tasks. Here we discuss several approximation algorithms for the least-squares support vector machine classifier. The proposed methods are based on randomized block kernel matrices, and we show that t…
Study shows how SGD's implicit regularization relates to ridge regression.
problem Least squares regression optimization with mini-batch SGD.
method Analyzes stochastic gradient flow as a continuous-time model of SGD.
result Bound on excess risk of SGD flow over ridge regression, revealing how parameters drive risk.
Regularized least-squares (kernel-ridge / Gaussian process) regression is a fundamental algorithm of statistics and machine learning. Because generic algorithms for the exact solution have cubic complexity in the number of datapoints, large datasets require to resort to approximations. In this work, the computation of …
Reinforced optimal controlmath.OC
Improved reinforcement method for optimal control problems.
problem Optimal control problems with limited computational cost.
method Reinforced least squares Monte Carlo method for stochastic control problems.
result Significant improvement in method's efficiency and accuracy.
Consider Least Squares Monte Carlo (LSM) algorithm, which is proposed by Longstaff and Schwartz (2001) for pricing American style securities. This algorithm is based on the projection of the value of continuation onto a certain set of basis functions via the least squares problem. We analyze the stability of the algori…
We propose a new forward-backward stochastic differential equation solver for high-dimensional derivatives pricing problems by combining deep learning solver with least square regression technique widely used in the least square Monte Carlo method for the valuation of American options. Our numerical experiments demonst…
Speeds up complex portfolio exposure calculations.
problem Calculating exposure of portfolios with exotic derivatives.
method Least Squares Monte Carlo (LSMC) technique.
result Significantly reduces computation time for nested Monte Carlo.
Estimates time-varying parameters from two OLS estimates.
problem Time-varying linear regression with hidden dynamics.
method Combines two OLS estimates for stable linear dynamics.
result Finite sample guarantee on estimation error.
This paper studies statistical estimation in optional regression models.
problem Estimating parameters in regression models with optional semimartingale processes.
method Structural least squares (LS) estimates and their sequential versions.
result Strong consistency of LS-estimates and fixed accuracy of sequential LS-estimates.
Dual-sPLS improves feature selection and prediction in high-dimensional data.
problem Relating variables to a response in high-dimensional chemometric problems.
method Generalizes PLS1 algorithm with dual norm penalizations and a shrinking ratio parameter.
result Favorably compares to similar regression methods on simulated and real chemical data.
We develop efficient algorithms to estimate the stability of Ordinary Least Squares regression results.
problem Measuring the stability of regression conclusions in low dimensions.
method Efficient algorithms for estimating the minimum number of samples that need to be removed to change a regression conclusion.
result We can estimate stability up to a factor of 3 better than the greedy heuristic and certify stability even for dropping a majority of samples.
Riesz regression connects to density ratio estimation for causal inference.
problem Estimating average treatment effects in causal inference.
method Riesz regression as a signed density ratio and least-squares importance fitting.
result Riesz regression and DRE are equivalent, allowing transfer of DRE results.
We prove a new and general concentration inequality for the excess risk in least-squares regression with random design and heteroscedastic noise. No specific structure is required on the model, except the existence of a suitable function that controls the local suprema of the empirical process. So far, only the case of…