ESNs trained with Tikhonov least squares approximate ergodic dynamical systems in L2(μ) norm.
problem Approximating ergodic dynamical systems using ESNs.
method Tikhonov least squares regression on ESNs trained on observations from an ergodic dynamical system.
result ESNs trained with Tikhonov least squares approximate the target function in the L2(μ) norm.
Unified approach to compute asymptotic constants using optimization.
problem Computing unknown constants in asymptotic expansions.
method Linear Least Squares and Tikhonov Linear Least Squares methods.
result Rigorous asymptotic estimates and convergence-rate guarantees.
Improved regression analysis using Padé approximants with new residuals and regularization.
problem Improving regression analysis with Padé approximants for accuracy and avoiding overfitting.
method New residuals in least squares method, system of linear equations for rational functions, Tikhonov regularization.
result Demonstrated efficiency in practical cases from physics and reliability theory.
Paper improves learning rates for GSC loss functions using iterated Tikhonov regularization.
problem Improving learning rates for GSC loss functions.
method Iterated Tikhonov regularization using proximal point method.
result Achieves fast and optimal rates for GSC loss functions.
New learning rates derived for Tikhonov-regularized problems without kernel assumptions.
problem Learning rates for Tikhonov-regularized learning problems.
method Minimax adaptive rates derived using Fourier isocapacitary condition and interpolation theory.
result Derivation of minimax adaptive rates without requiring kernel assumptions.
We find the optimal Tikhonov regularizer for linear inverse problems without prior knowledge.
problem Finding the optimal regularizer for linear inverse problems in imaging.
method Characterization of the optimal regularizer and learning from data.
result The optimal regularizer is independent of the forward operator and depends only on the mean and covariance of the random variable.
Optimal tuning of Tikhonov regularizers achieves best performance without additional cost.
problem Selecting the best estimator among Tikhonov regularized estimators or their linear combinations.
method Convex aggregation procedure based on Q Q Q -aggregation. result Error term does not depend on penalty matrix or number of estimators.
Study Tikhonov regularization for non-linear inverse problems to improve image reconstruction accuracy.
problem Reconstructing quantities from noisy, non-linearly transformed observations.
method Tikhonov regularization using reproducing kernel Hilbert spaces.
result Developed optimal convergence rates for the estimator.
Illustrates interleaved learning with Kalman Filter for linear least squares.
problem Improving machine learning algorithms through interleaved learning.
method Simple statistical and optimization framework based on Kalman Filter.
result Demonstrates the effectiveness of interleaved learning.
CD converges linearly for MCP/SCAD penalized least squares.
problem Recovering sparse signals from data.
method Coordinate descent for MCP/SCAD penalized least squares.
result CD converges linearly to solutions of MCP/SCAD penalized least squares.
The paper improves Kaczmarz algorithm with momentum for linear least squares.
problem Improving convergence of the Kaczmarz algorithm for linear least squares.
method Integrates geometrically smoothed momentum into the randomized Kaczmarz algorithm.
result Proves expected error reduction in singular vector directions.
Unified framework recovers exact input from SOM activation patterns.
problem Generating high-dimensional data from Self-Organizing Maps (SOMs).
method Inverting SOM activation patterns to recover input, using linear system and Tikhonov regularization.
result MUSIC framework produces coherent semantic transitions and maintains high classifier confidence.
Paper introduces a new IV regression method for mixed-frequency data.
problem Estimating high-dimensional slope parameters in mixed-frequency data.
method Tikhonov-regularized estimator for high-dimensional linear IV regression.
result High-dimensional slope parameter can be accurately estimated using a low-frequency instrumental variable.
This book introduces linear models and their theories rigorously.
problem Understanding linear models and their theories.
method Explains linear models from three perspectives, introduces maximum likelihood estimation, and proves least squares is the best unbiased linear model.
result Least squares is the best unbiased linear model in terms of mean squared error.
Least squares estimator fails to achieve optimal risk in bounded distributions, but non-linear predictors can.
problem Optimal risk in bounded distributions for constrained least squares.
method Comparison of least squares and non-linear predictors.
result Non-linear predictors can achieve optimal risk O ( d / n ) O(d/n) O ( d / n ) in bounded distributions. Sparse linear regression, which entails finding a sparse solution to an underdetermined system of linear equations, can formally be expressed as an l 0 l_0 l 0 -constrained least-squares problem. The Orthogonal Least-Squares (OLS) algorithm sequentially selects the features (i.e., columns of the coefficient matrix) to greedil…
A new method for estimating large-scale linear models with improved precision.
problem Estimating large-scale linear statistical models efficiently.
method Sequential Least-Squares Estimators with Fast Randomized Sketching (SLSE-FRS), integrating Sketch-and-Solve and Iterative-Sketching methods.
result SLSE-FRS produces high-precision estimators, outperforming state-of-the-art methods.
A new algorithm solves nonnegative least squares faster with nonnegative data.
problem Nonnegative least squares problems with nonnegative data.
method Primal-dual perspective accelerated algorithm with adaptive restart.
result Oracle complexity independent of matrix constants, solvable to multiplicative error.
Study identifies and validates a method for system identification of Markov jump linear systems.
problem System identification for autonomous Markov jump linear systems with complete state observations.
method Proposes switched least squares method for identification and derives rates of convergence.
result Data-independent rate of convergence is O ( log ( T ) / T ) \mathcal{O}\big(\sqrt{\log(T)/T} \big) O ( log ( T ) / T ) , showing strong consistency. Unified analysis of reweighted least-squares algorithms for linear models.
problem Recovering unknown signals from linear measurements using reweighted least squares.
method Unified asymptotic analysis of IRLS, lin-RFM, and alternating minimization algorithms.
result The algorithms can achieve favorable performance in a few iterations with appropriate reweighting.
Correcting bias in least squares regression with volume-rescaled sampling.
problem Bias in linear least squares solutions without distributional assumptions.
method Volume-rescaled sampling to correct bias in i.i.d. samples.
result Combined sample becomes unbiased with rescaled volume additional sample.
Optimal multiscale learning of linear operators
problem Statistical and computational limits of learning bounded linear operators between Sobolev spaces
method Reformulate as an infinite-dimensional matrix regression problem with heterogeneous multiscale structure
result Establish minimax rates and construct a finite-resolution blockwise least-squares estimator attaining these rates
This paper reviews and compares supervised linear dimension-reduction techniques.
problem Lack of information in the response during unsupervised PCA reduces predictive performance.
method Review and comparison of supervised linear dimension-reduction techniques.
result PLS and LSPCA consistently outperform other techniques in simulations.
New algorithm estimates partially-observed linear systems with better rates than previous methods.
problem Estimating parameters of partially-observed linear systems with long-term dependencies and semi-parametric noise.
method Prefiltered least squares estimator with semi-parametric noise model.
result First algorithm provably estimates parameters of partially-observed linear systems with rates not dependent on dependency decay rate.
A set of introductory notes on the subject of data classification using a linear classifier and least-squares cost function, and the negative effect of the presence of outliers on the decision boundary of the linear discriminant. We also show how a simple scaling could make the outlier less significant, thereby obtaini…
The paper analyzes the risk of a least squares estimator under a spike covariance model.
problem Risk analysis of the least squares estimator under a spike covariance model.
method Assumes spike covariance matrices, studies risk as d / n i g h t a r r o w ∞ d/n
ightarrow \infty d / ni g h t a r r o w ∞ . result Risk of the minimum norm least squares estimator vanishes compared to the null estimator.
Paper proposes a pre-conditioning technique to speed up gradient-descent convergence in distributed linear least-squares problems.
problem Expediting convergence of gradient-descent method for ill-conditioned distributed linear least-squares problems.
method Iterative pre-conditioning technique to improve convergence rate of gradient-descent method.
result Pre-conditioned gradient-descent achieves superlinear convergence for unique solutions and improved linear convergence otherwise.
New framework assesses regularization norms in ill-posed problems, revealing L2 instability and proposing adaptive fractional RKHS solutions.
problem Comparative analysis of regularization norms in ill-posed problems.
method Small noise analysis framework for Tikhonov and RKHS regularizations.
result Optimal convergence rates achieved with adaptive fractional RKHS, but hyper-parameters decay too fast.
Kaczmarz++ accelerates convergence for ill-conditioned systems.
problem Solving ill-conditioned linear systems efficiently.
method Adaptive momentum acceleration, Tikhonov-regularized projections, and memoization.
result Kaczmarz++ converges faster than Krylov methods on ill-conditioned systems.
Ordinary least squares (OLS) is the default method for fitting linear models, but is not applicable for problems with dimensionality larger than the sample size. For these problems, we advocate the use of a generalized version of OLS motivated by ridge regression, and propose two novel three-step algorithms involving l…
The kernel least mean squares (KLMS) algorithm is a computationally efficient nonlinear adaptive filtering method that "kernelizes" the celebrated (linear) least mean squares algorithm. We demonstrate that the least mean squares algorithm is closely related to the Kalman filtering, and thus, the KLMS can be interpreted…
This paper optimizes sampling for least-squares approximation.
problem Optimizing sampling for least-squares approximation in arbitrary linear spaces.
method Introducing the Christoffel function to construct near-optimal random sampling strategies.
result The number of samples scales log-linearly in the dimension of the approximation space.
Paper introduces ℓ \ell ℓ -DER for regression tasks using morphological operators and convex-concave procedure.
problem Developing a universal approximator for regression tasks.
method Introduces ℓ \ell ℓ -DER model, trains it using a convex-concave procedure (CCP) to minimize least-squares. result Outperforms other hybrid morphological models and state-of-the-art approaches.
The family of temporal difference (TD) methods span a spectrum from computationally frugal linear methods like TD(λ) to data efficient least squares methods. Least square methods make the best use of available data directly computing the TD solution and thus do not require tuning a typically highly sensitive learning r…
Algorithm solves robust linear regression with block Lewis weights.
problem Group distributionally robust least squares problem.
method Algorithm based on geometric construction and block Lewis weights, using accelerated proximal methods.
result Improves over known methods for moderate accuracy regimes and matches state-of-the-art guarantees.
The least-squares support vector machine is a frequently used kernel method for non-linear regression and classification tasks. Here we discuss several approximation algorithms for the least-squares support vector machine classifier. The proposed methods are based on randomized block kernel matrices, and we show that t…
The OLS estimator optimally identifies stable linear systems with a finite number of samples.
problem Identifying stable linear systems with a finite number of samples.
method Finite-time analysis of the Ordinary Least Squares (OLS) estimator for stable linear systems.
result The OLS estimator achieves optimal sample complexity for stable systems, matching existing lower bounds up to universal factors.
Lecture notes on advanced linear regression methods.
problem Understanding the properties of linear regression estimators in high dimensions.
method Proposition-proof exploration of least squares, ridgeless, ridge, and lasso estimators.
result Detailed analysis of the existence, uniqueness, relations, computation, and non-asymptotic properties of these estimators.
New insights into learning for blind inverse problems with theoretical guarantees.
problem Learning in blind inverse problems where both signal and operator are unknown.
method Data-driven approaches using Linear Minimum Mean Square Estimators (LMMSEs) with theoretical analysis.
result Established equivalences with Tikhonov-regularized formulations and derived finite-sample error bounds.
The paper explores how overfitting can lead to better predictions in high-dimensional data.
problem Understanding the behavior of linear models in high-dimensional settings with more predictors than observations.
method Analysis of ordinary least squares, penalized least squares, and spectral shrinkage estimates.
result The phenomenon of double descent, where model performance can improve with increasing model complexity.
A new method reduces the cost of solving large-scale linear models.
problem Solving large-scale linear statistical models efficiently.
method Sequential Preconditioned Conjugate Gradient Method (SPCG).
result SPCG achieves OLS prediction accuracy with fewer iterations and less time.
Optimal hashing embeddings reduce linear least squares solving time.
problem Efficiently solving large-scale linear least squares problems.
method Optimal hashing sketching matrices for linear least squares.
result Ski-LLS outperforms state-of-the-art solvers on various problem types.
Least-squares models such as linear regression and Linear Discriminant Analysis (LDA) are amongst the most popular statistical learning techniques. However, since their computation time increases cubically with the number of features, they are inefficient in high-dimensional neuroimaging datasets. Fortunately, for k-fo…
New method for estimating parameters in inverse problems using double robustness.
problem Estimating parameters defined as linear functionals of solutions to linear inverse problems.
method Source condition double robust inference method that uses iterated Tikhonov regularized adversarial estimators.
result Asymptotic normality of the parameter of interest as long as either the primal or dual inverse problem is sufficiently well-posed.
Estimates time-varying parameters from two OLS estimates.
problem Time-varying linear regression with hidden dynamics.
method Combines two OLS estimates for stable linear dynamics.
result Finite sample guarantee on estimation error.
This study examines the relationship between PLS and OLS regression using eigenvalue distributions.
problem Analyzing the difference between PLS and OLS regression in terms of eigenvalue distributions.
method Examined the distance between PLS and OLS regression coefficients using the Mahalanobis distance and eigenvalue distributions of the regressor covariance matrix.
result Provided a bound on the distance between PLS and OLS regression coefficients that depends only on the eigenvalue distribution of the regressor covariance matrix.
New algorithm for online collaborative filtering using linear bandits and alternating least squares.
problem Online collaborative filtering with item recommendations over time.
method Combines linear bandits and alternating least squares for matrix factorization.
result Superior performance in cumulative regret and average cumulative NDCG over state-of-the-art algorithms.
Efficiently estimates private least squares with linear error growth.
problem Private estimation of ordinary least squares with bounded residuals and leverage.
method Scaled noise added to a stable nonprivate estimator of the regression vector.
result Near-optimal accuracy guarantee with linear error growth in dimension.