We compare the risk of ridge regression to a simple variant of ordinary least squares, in which one simply projects the data onto a finite dimensional subspace (as specified by a Principal Component Analysis) and then performs an ordinary (un-regularized) least squares regression in this subspace. This note shows that …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,181 papers · 148 categories
Trend · papers per month
48 results for “ordinary least-squares”
Cross validation residuals extended to GLS models.
problem Validating models with correlated data.
method Leave-M-out cross validation for GLS models, demonstrating relationship with Cook's distance.
result No need to refit model for reduced datasets.
This paper proposes a new method to prevent backtesting overfitting in trading strategies.
problem Preventing misleading results in backtesting of trading strategies.
method Covariance-Penalty Correction approach to reduce risk metrics based on the number of parameters and data used.
result Covariance-Penalties are effective in avoiding backtesting overfitting, with Total Least Squares outperforming Ordinary Least Squares.
We develop efficient algorithms to estimate the stability of Ordinary Least Squares regression results.
problem Measuring the stability of regression conclusions in low dimensions.
method Efficient algorithms for estimating the minimum number of samples that need to be removed to change a regression conclusion.
result We can estimate stability up to a factor of 3 better than the greedy heuristic and certify stability even for dropping a majority of samples.
Estimates time-varying parameters from two OLS estimates.
problem Time-varying linear regression with hidden dynamics.
method Combines two OLS estimates for stable linear dynamics.
result Finite sample guarantee on estimation error.
OLS is a special case of Transformer, revealing its linear nature.
problem Understanding the statistical essence of Transformer architecture.
method Algebraic proof and spectral decomposition of covariance matrix.
result Attention mechanism in Transformers is mathematically equivalent to OLS.
Ensemble methods improve model performance by averaging over subsampled predictors.
problem Understanding the effect of feature subsampling in ensemble methods.
method Fit linear predictors using ordinary least squares on random submatrices of the data matrix.
result The asymptotic risk of an ensemble is equal to the ridge regression risk, optimal for linear predictors.
OLS predictions are shown to be similar to attention mechanisms in models.
problem OLS in traditional statistics and econometrics.
method Rewriting OLS as an attention mechanism in a transformed space.
result OLS can be understood as minimizing squared prediction errors via optimal embedding and decoding.
A new method estimates parameters of complex models using ordinary least squares.
problem Estimating parameters of nonlinear dynamic models from time series data.
method Physics-Informed Regression (PIR) using regularized ordinary least squares.
result PIR outperforms physics-informed neural networks (PINN) in parameter estimation.
This book introduces linear models and their theories rigorously.
problem Understanding linear models and their theories.
method Explains linear models from three perspectives, introduces maximum likelihood estimation, and proves least squares is the best unbiased linear model.
result Least squares is the best unbiased linear model in terms of mean squared error.
Finite-time Identification of Stable Linear Systems: Optimality of the Least-Squares Estimatormath.ST
The OLS estimator optimally identifies stable linear systems with a finite number of samples.
problem Identifying stable linear systems with a finite number of samples.
method Finite-time analysis of the Ordinary Least Squares (OLS) estimator for stable linear systems.
result The OLS estimator achieves optimal sample complexity for stable systems, matching existing lower bounds up to universal factors.
Ordinary least squares (OLS) is the default method for fitting linear models, but is not applicable for problems with dimensionality larger than the sample size. For these problems, we advocate the use of a generalized version of OLS motivated by ridge regression, and propose two novel three-step algorithms involving l…
Optimal noise excitation for linear system identification reduces sample complexity.
problem Efficiently identifying linear systems with minimal data.
method Active learning algorithm using ordinary least squares and semidefinite programming.
result The proposed algorithm matches lower bounds on sample complexity for any active learning method.
Rational Krigingstat.ME
New kriging method improves mean estimation and uncertainty.
problem Improving mean estimation and uncertainty in kriging.
method Proposes a rational kriging method using generalized least squares and Gaussian process.
result Generalized least squares estimate is more well-behaved than ordinary kriging.
Efficiently estimates private least squares with linear error growth.
problem Private estimation of ordinary least squares with bounded residuals and leverage.
method Scaled noise added to a stable nonprivate estimator of the regression vector.
result Near-optimal accuracy guarantee with linear error growth in dimension.
A joint optimization approach to identifying sparse dynamics using least squares kernel collocationstat.ME
Develops a new method for learning ODEs from sparse data.
problem Learning systems of ODEs from scarce, partial, and noisy data.
method Combines sparse recovery and RKHS techniques.
result Significant gains in accuracy, sample efficiency, and robustness to noise.
This study examines the relationship between PLS and OLS regression using eigenvalue distributions.
problem Analyzing the difference between PLS and OLS regression in terms of eigenvalue distributions.
method Examined the distance between PLS and OLS regression coefficients using the Mahalanobis distance and eigenvalue distributions of the regressor covariance matrix.
result Provided a bound on the distance between PLS and OLS regression coefficients that depends only on the eigenvalue distribution of the regressor covariance matrix.
The paper explores how overfitting can lead to better predictions in high-dimensional data.
problem Understanding the behavior of linear models in high-dimensional settings with more predictors than observations.
method Analysis of ordinary least squares, penalized least squares, and spectral shrinkage estimates.
result The phenomenon of double descent, where model performance can improve with increasing model complexity.
BCDP enhances privacy by protecting sensitive features more precisely.
problem Uniform privacy protection in LDP degrades performance for sensitive features.
method Bayesian Coordinate Differential Privacy (BCDP) adjusts privacy protection per feature sensitivity.
result BCDP improves accuracy in downstream tasks without sacrificing privacy.
OLS estimator nearly optimally identifies linear systems from single trajectory.
problem Identifying linear dynamical systems from a single observed trajectory.
method Generalized small-ball method for dependent data, avoiding mixing-time arguments.
result OLS estimator nearly matches minimax optimal performance for linear systems.
This paper reviews SDR methods for multivariate response regression.
problem Handling sufficient dimension reduction for multivariate response regression.
method Characterizes SDR estimators as inverse or forward regression methods.
result Pooled marginal, projective resampling, distance-based, ordinary least squares, partial least squares, and semiparametric SDR estimators are discussed.
Variable selection is a challenging issue in statistical applications when the number of predictors far exceeds the number of observations . In this ultra-high dimensional setting, the sure independence screening (SIS) procedure was introduced to significantly reduce the dimensionality by preserving the true mod…
Subsampling methods have been recently proposed to speed up least squares estimation in large scale settings. However, these algorithms are typically not robust to outliers or corruptions in the observed covariates. The concept of influence that was developed for regression diagnostics can be used to detect such corrup…
R2T hybrid model improves robust regression for asymmetric noise.
problem Least-squares regression fails with asymmetric structured noise.
method Transformer encoder, compression NN, fixed symbolic equation.
result Median regression MSE of 6e-6 to 3.5e-5 on synthetic data.
A new method reduces the cost of solving large-scale linear models.
problem Solving large-scale linear statistical models efficiently.
method Sequential Preconditioned Conjugate Gradient Method (SPCG).
result SPCG achieves OLS prediction accuracy with fewer iterations and less time.
A number of recent emerging applications call for studying data streams, potentially infinite flows of information updated in real-time. When multiple co-evolving data streams are observed, an important task is to determine how these streams depend on each other, accounting for dynamic dependence patterns without impos…
New algorithms protect user data while optimizing personalized decisions.
problem Personalized decision-making with private user data.
method Developed LDP algorithms for stochastic generalized linear bandits using SGD and OLS.
result Achieved the same regret bound as non-privacy settings with LDP.
We develop a method to efficiently solve tensor regression problems with significantly reduced dimensions.
problem Efficiently solving tensor regression problems with reduced dimensions.
method Sparse random projections to reduce tensor dimensions and apply ordinary least squares regression.
result Near-optimal solutions to the reduced problem imply near-optimal solutions to the original tensor regression problem.
The paper explores machine learning methods for proxy modeling in life insurance solvency capital requirements.
problem Life insurance companies need to estimate solvency capital requirements from full loss distributions, but computational limitations restrict full simulations.
method The paper presents various adaptive machine learning approaches to approximate the risk-dependent proxy function using least-squares Monte Carlo.
result The machine learning methods significantly improve the accuracy and efficiency of proxy modeling compared to traditional regression techniques.
Study ablated data augmentation techniques and their mathematical equivalence to penalties.
problem Lack of mathematical understanding of differences between ablated data augmentation techniques.
method Formal model of mean ablated data augmentation and inverted dropout for linear regression; empirical validation for deep networks.
result Ablated data augmentation and inverted dropout are mathematically equivalent to penalties in optimization.
Cross-validation estimates model performance on unseen data, not training data.
problem Understanding how cross-validation estimates prediction error and its limitations.
method Analyzing linear models and popular prediction error estimates, introducing nested cross-validation.
result Cross-validation estimates the average prediction error of models fit on other unseen training sets, not the model at hand.
WENDy now estimates nonlinear ODEs with noisy data.
problem Estimating parameters of nonlinear ODEs with noisy data.
method WENDy-MLE algorithm for maximum likelihood estimation of nonlinear-in-parameters ODEs.
result WENDy-MLE outperforms other methods in accuracy, speed, and domain of convergence.
Two algorithms improve fitting autoregressive models for big data.
problem Efficiently solving Toeplitz least squares problems for large time series data.
method Applied randomized numerical linear algebra (RandNLA) techniques.
result LSAR algorithm is more robust for real-world time series data.
Convex Total Least Squaresstat.ML
We study the total least squares (TLS) problem that generalizes least squares regression by allowing measurement errors in both dependent and independent variables. TLS is widely used in applied fields including computer vision, system identification and econometrics. The special case when all dependent and independent…
Improved privacy-preserving linear regression via iterative Hessian mixing.
problem Differentially private linear regression with improved accuracy and efficiency.
method Iterative Hessian Mixing (IHM) for differentially private ordinary least squares (DP-OLS).
result IHM provides better utility guarantees and outperforms AdaSSP in empirical evaluations.
We consider the problem of predicting as well as the best linear combination of d given functions in least squares regression, and variants of this problem including constraints on the parameters of the linear combination. When the input distribution is known, there already exists an algorithm having an expected excess…
Model Selection for High-Dimensional Regression under the Generalized Irrepresentability Conditionmath.ST
In the high-dimensional regression model a response variable is linearly related to covariates, but the sample size is smaller than . We assume that only a small subset of covariates is `active' (i.e., the corresponding coefficients are non-zero), and consider the model-selection problem of identifying the a…
Improves regression models' performance on covariate shift.
problem Out-of-distribution generalization for regression.
method Spectrally adapting the weights of a pre-trained neural regression model.
result Spectral adaptation improves out-of-distribution performance.
From STLS to Projection-based Dictionary Selection in Sparse Regression for System Identificationstat.ML
This work improves SINDy-type algorithms for system identification using score-guided dictionary selection.
problem Improving accuracy and interpretability in dynamical system identification.
method Score-guided library selection to refine dictionary terms in sparse regression.
result Score-guided methods enhance SINDy's robustness in discovering governing equations.
We propose a general semi-supervised inference framework focused on the estimation of the population mean. As usual in semi-supervised settings, there exists an unlabeled sample of covariate vectors and a labeled sample consisting of covariate vectors along with real-valued responses ("labels"). Otherwise, the formulat…
Biased mean regression estimates factors exceeding expected loss or radiation release severity.
problem Estimating factors exceeding expected loss or radiation severity levels.
method Biased mean regression using superexpectation error minimization.
result Equivalent to quantile regression and CVaR optimization under specific conditions.
Transformers learn a mesa-optimizer to implement in-context learning.
problem Understanding the convergence of autoregressive training to a mesa-optimizer.
method Investigated a one-layer linear causal self-attention model autoregressively trained by gradient flow.
result Proved that autoregressive training converges to a gradient descent step for an OLS problem, validating the mesa-optimizer hypothesis.
Proposes a novel linear discriminant analysis for matrix-valued data.
problem Classification of high-dimensional matrix-valued data from imaging studies.
method Efficient nuclear norm penalized regression with low-rank structure.
result Superior performance compared to existing methods in simulations and EEG data.
This work gives a simultaneous analysis of both the ordinary least squares estimator and the ridge regression estimator in the random design setting under mild assumptions on the covariate/response distributions. In particular, the analysis provides sharp results on the ``out-of-sample'' prediction error, as opposed to…
Paper proposes a debiased estimator for adaptive linear regression.
problem Non-normal asymptotic behavior of OLS estimator in adaptive linear regression.
method Adaptive linear estimating equations to construct debiased estimator.
result Established asymptotic normality of the debiased estimator.
The paper provides risk bounds for learning many response functions using linear regression.
problem Learning many response functions from a single dataset.
method Ordinary least squares regression in a high-dimensional feature space.
result Convergence guarantees on worst-case excess prediction risk for infinite response functions with finite VC dimension.
The paper learns an autoregressive filter for unknown dynamical systems with robust guarantees.
problem Learning optimal predictions in an unknown dynamical system.
method Directly learns an autoregressive filter using an -based objective, regressing on both inputs and outputs.
result The algorithm has optimal sample complexity in terms of the rollout length.
This paper compares TFC and LS-SVM for solving DEs, showing TFC is faster and more accurate.
problem Solving differential equations efficiently and accurately.
method Theory of Functional Connections (TFC) integrated with LS-SVM.
result TFC is faster and more accurate than LS-SVM for solving DEs.