HARFE approximates sparse additive functions using random features and ridge regression.
problem Approximating high-dimensional sparse additive functions.
method Hard-ridge random feature expansion with sparse ridge regression and hard-thresholding pursuit.
result HARFE method converges with a given error bound and achieves lower error than other algorithms.
OKRidge solves sparse ridge regression problems for nonlinear systems.
problem Identifying sparse governing equations for nonlinear dynamical systems.
method OKRidge algorithm using saddle point formulation and ADMM-based approach with efficient proximal operators.
result OKRidge achieves provable optimality with significantly faster run times than Gurobi.
Efficiently performs robust and sparse kernel regression.
problem Robust and sparse kernel regression.
method Sign gradient descent and early stopping.
result Sign gradient descent achieves robust and sparse kernel regression efficiently.
The paper proposes a gradient-based method for multi-penalty Ridge regression.
problem Optimizing multiple regularization hyperparameters for linear regression.
method Gradient-based optimization through matrix differential calculus.
result The method outperforms traditional regularization techniques like LASSO and Ridge.
Bayesian EM method improves ridge regression tuning without LOOCV's limitations.
problem Tuning ridge regression hyper-parameter λ λ λ efficiently and accurately. method Bayesian formulation of ridge regression with expectation maximization (EM) procedure.
result Guaranteed unique optimal solution for large n n n without candidate λ λ λ specification. When the design matrix has orthonormal columns, "soft thresholding" the ordinary least squares (OLS) solution produces the Lasso solution [Tibshirani, 1996]. If one uses the Puffer preconditioned Lasso [Jia and Rohe, 2012], then this result generalizes from orthonormal designs to full rank designs (Theorem 1). Theorem …
New algorithm tackles high-dimensional simulation optimization, converging efficiently.
problem High-dimensional simulation optimization challenges.
method Sparse grid experimental design combined with kernel ridge regression using Brownian field kernel, followed by expected improvement strategy.
result Established upper bounds on convergence rate, demonstrating superior performance in practice.
New method identifies network structure without regularization for sparse teacher couplings.
problem Identifying network structure in inverse Ising problems with model mismatch.
method Ridge linear regression with two-stage estimator.
result Perfect identification of network structure possible without regularization for sparse teacher couplings.
A novel approach tackles sparse linear bandits with reduced communication costs and minimal cumulative regret.
problem Sparse linear bandits with high-dimensional feature vectors and limited relevant features.
method Cooperative Thresholded Lasso using Lasso and ridge regression for dimension reduction and aggregation.
result Regret bound of O ( s 0 log d + s 0 T ) \mathcal{O}(s_0 \log d + s_0 \sqrt{T}) O ( s 0 log d + s 0 T ) with high probability. Improves test set performance and reduces out-of-sample disappointment for unstable models.
problem Ensuring strong test set performance via cross-validation for unstable models.
method Nested k-fold cross-validation with hyperparameter selection based on a weighted sum of cross-validation metric and model stability measure.
result Improves out-of-sample MSE for sparse ridge regression and CART by 4% and 2% respectively, compared to k-fold cross-validation.
Boosting ridge regression for high-dimensional data classification reduces computational cost and improves learning time.
problem High computational demand of inverting regularised covariance matrix in ridge regression for high-dimensional problems.
method Train an ensemble of ridge regressors in randomly projected subspaces, then combine them using adaptive boosting.
result Effective in terms of learning time and improved predictive performance in some cases.
Short proof shows how ridge regression works with random data.
problem Understanding prediction error in ridge regression with random design.
method Combination of exchangeability arguments, matrix perturbation, and operator convexity.
result Elementary proof of prediction error without complex inequalities.
Kernel ridge regression imputation with consistent variance estimation for handling missing data.
problem Handling missing data in statistical analysis.
method Kernel ridge regression imputation combined with entropy method for variance estimation.
result Root-n consistency of the imputation estimator in a Sobolev space setting.
Novel confidence intervals improve convergence rates for sparse kernel-based models.
problem High computational cost in kernel-based learning models.
method Novel confidence intervals for Nyström method and sparse variational Gaussian process approximation.
result Improved performance bounds in regression and optimization problems.
Prevalidated ridge regression simplifies logistic regression for high-dimensional data.
problem Efficient probabilistic classification in high-dimensional data with logistic regression.
method Developed a prevalidated ridge regression model that matches logistic regression's performance but is more computationally efficient.
result Prevalidated ridge regression achieves similar classification error and log-loss to logistic regression for high-dimensional data.
Proposes a new ridge estimator for smooth covariates with adaptive centering.
problem Estimating coefficients and center function for smooth covariates in linear models.
method SACR framework with convex formulation, roughness penalty, and adaptive centering.
result Improves prediction and variable selection for smooth covariates.
Derives ideal train/test split for ridge regression in large data limit.
problem Finding optimal train/test split for ridge regression in large data scenarios.
method Mathematical derivation of optimal train/test split, considering ridge tuning parameter and asymptotic behavior.
result The optimal train/test split for ridge regression in the large data limit depends weakly on the ridge tuning parameter alpha.
Novel algorithm identifies nonlinear Granger causal relationships using kernel ridge regression.
problem Identification of nonlinear Granger causal relationships.
method Flexible plug-in architecture with kernel ridge regression using radial basis function.
result Kernel ridge regression in mlcausality achieves competitive AUC scores and more finely calibrated p-values.
GD outperforms ridge regression and SGD in linear regression problems.
problem Comparing the risks of GD, ridge regression, and SGD in linear regression problems.
method Instance-wise finite-sample risk analysis of GD, ridge regression, and SGD.
result GD outperforms ridge regression and is incomparable with SGD in some cases.
We study the performance of sparse regression methods and propose new techniques to distill the governing equations of dynamical systems from data. We first look at the generic methodology of learning interpretable equation forms from data, proposed by Brunton et al., followed by performance of LASSO for this purpose. …
Paper proposes methods for transfer learning with random coefficient ridge regression.
problem Estimation and prediction in high-dimensional settings with related models.
method Two estimators using weighted sums of ridge estimates from target and source models.
result Explicit expression of estimation and prediction risks derived using random matrix theory.
SGD implicitly regularizes linear regression problems better than ridge regression for many cases.
problem Understanding implicit regularization in linear regression problems.
method Comparing SGD and ridge regression on a broad class of least squares problems.
result SGD generalizes no worse than ridge regression for many problem instances, sometimes better.
Proposes ridge regression on Riemannian manifolds for time-series prediction.
problem Time-series prediction on Riemannian manifolds.
method Combines Riemannian least-squares fitting via Bézier curves, empirical covariance on manifolds, and Mahalanobis distance regularization.
result Significant error reduction in synthetic spherical experiments and hurricane forecasting.
Gradient descent outperforms ridge regression under certain covariance matrix decay conditions.
problem Comparing the performance of gradient descent and ridge regression in linear models.
method Investigated gradient descent and ridge regression for linear regression with random isotropic ground truth.
result Gradient descent outperforms ridge regression under specific covariance matrix decay conditions.
Ridge regression is revisited with debiasing and thresholding, offering advantages over Lasso.
problem High-dimensional data challenges classical ridge regression's sparsity detection and bias issues.
method Debiasing and thresholding ridge regression, introducing a wild bootstrap for confidence regions and hypothesis testing, and a hybrid bootstrap for prediction intervals.
result Debiased and thresholded ridge regression can offer similar performance to thresholded Lasso and may be preferable in some settings.
We study a seemingly unexpected and relatively less understood overfitting aspect of a fundamental tool in sparse linear modeling - best subset selection, which minimizes the residual sum of squares subject to a constraint on the number of nonzero coefficients. While the best subset selection procedure is often perceiv…
Kernel ridge regression inference for nonstandard data.
problem Inferential theory for kernel ridge regression with nonstandard data.
method Constructs valid and sharp confidence sets using anti-symmetric multipliers.
result Develops a test for match effects in school matching mechanisms.
pGMM kernel outperforms ordinary ridge regression and RBF kernel ridge regression without tuning.
problem Comparing pGMM kernel regression with other ridge regression methods.
method Implemented and compared pGMM kernel regression with ordinary ridge regression and RBF kernel ridge regression.
result pGMM kernel performs well without tuning and can match boosted trees with parameter tuning.
Ridge leverage scores provide a balance between low-rank approximation and regularization, and are ubiquitous in randomized linear algebra and machine learning. Deterministic algorithms are also of interest in the moderately big data regime, because deterministic algorithms provide interpretability to the practitioner …
We compare the risk of ridge regression to a simple variant of ordinary least squares, in which one simply projects the data onto a finite dimensional subspace (as specified by a Principal Component Analysis) and then performs an ordinary (un-regularized) least squares regression in this subspace. This note shows that …
Localized sketching improves matrix multiplication and ridge regression complexity.
problem Efficiently approximate matrix multiplication and ridge regression with limited data availability.
method Localized sketching matrices for block diagonal structure, reducing sample complexity.
result Localized sketching achieves sample complexity matching global sketching methods.
Improved ridge regression with Frequent Directions for large-scale tasks.
problem Improving performance of ridge regression for large-scale data.
method Combines Frequent Directions with iterative optimization schemes.
result Achieves high accuracy in estimating bias and variance for sketched ridge regression.
Improves kernel ridge regression by optimizing scale and feature parameters.
problem Kernel ridge regression with fixed kernel.
method Introduces a matrix parameter U to optimize scale and feature parameters.
result Solves a nonlinear variational problem to optimize U.
Time-varying parameters are shown to be ridge regressions, simplifying computations and tuning.
problem Capturing structural change in economic data.
method Ridge regression approach, including cross-validation for tuning, and extensions for sparsity and reduced-rank restrictions.
result The method efficiently estimates large numbers of time-varying parameters, demonstrated with Canadian monetary policy data.
A new method for high-dimensional functional regression reduces multicollinearity and improves interpretability.
problem Multicollinearity, overfitting, and interpretability in high-dimensional functional linear models.
method Partition-based functional ridge regression framework.
result Improved numerical stability and enhanced interpretability without explicit variable selection.
Proposes an efficient shrinkage path for ridge regression.
problem Ill-conditioned data in linear models.
method A new generalized ridge regression shrinkage path that minimizes MSE risk.
result The path is as short as possible while maintaining optimal trade-off.
MGD with early stopping tends to ridge regularization in least squares regression.
problem Characterizing the implicit regularization of MGD with early stopping.
method Continuous-time view of MGD (momentum gradient flow) and comparison with explicit ridge regularization.
result Under optimal tuning, the risk of MGF is no more than 1.54 times that of ridge.
Paper proves convergence rates for Gaussian kernel ridge regression.
problem Understanding convergence rates for Gaussian kernel ridge regression.
method Establishes polynomial convergence rates for KRR with fixed hyperparameters.
result First polynomial convergence rates for Gaussian kernel ridge regression.
A new method corrects bias in high-dimensional ridge regression.
problem Inherent bias in ridge regression limits statistical efficiency and scalability.
method Iterative bias correction strategy for p < n p < n p < n and Ridge-Screening method for p > n p > n p > n . result Valid inferences and asymptotic properties established for de-biased ridge estimators.
Study ridge regression for non-identically distributed data with varying variances.
problem Investigate high-dimensional regression with non-identical data variance.
method Propose a random effect model and use tools from random matrix theory.
result Highlight the double descent phenomenon in high-dimensional regression for certain variance profiles.
HAR regression improves performance on small datasets.
problem Small datasets with complex functions.
method Data-adaptive kernel ridge regression using tensor-product spline basis.
result Achieves n − 1 / 3 n^{-1/3} n − 1/3 convergence rate for right-continuous functions. Researchers compare different gradient methods for ridge regression, finding conjugate gradients have similar performance.
problem Comparing statistical properties of different gradient methods in ridge regression.
method Explicit non-standard error decomposition to bound prediction error of conjugate gradient iterates.
result Conjugate gradient iterates share optimality properties with gradient flow and ridge regression up to a constant factor.
DRE combines DNN with random feature regression for efficient neural network design.
problem Designing and training deep neural networks (DNN) efficiently and effectively.
method DRE architecture with two-layer neural networks, randomly drawn input and output weights trained with linear ridge regression.
result DRE outperforms state-of-the-art DNN in many data sets with lower computational cost.
Face recognition is the important field in machine learning and pattern recognition research area. It has a lot of applications in military, finance, public security, to name a few. In this paper, the combination of the tensor sparse PCA with the nearest-neighbor method (and with the kernel ridge regression method) wil…
Ridge regression shows different behaviors in binary classification with noisy labels.
problem Binary classification with noisy labels and anisotropic cluster distributions.
method Investigation of ridge regression behavior in overparameterized settings with label noise.
result Ridge regression exhibits qualitatively different behavior based on the scale of cluster mean vectors and covariance matrices.
A new method reparameterizes ridge regression for faster, more interpretable results.
problem Challenges in selecting hyperparameter α for ridge regression.
method Fractional Ridge Regression (FRR) reparameterizes RR in terms of the ratio γ.
result FRR solutions vary with different γ, avoiding wasted calculations and manual exploration.
ParK efficiently solves kernel ridge regression for large datasets.
problem Large-scale kernel ridge regression efficiency and accuracy.
method Partitioning feature space with random projections and iterative optimization.
result Provably maintains statistical accuracy with reduced space and time complexity.
We study the following three fundamental problems about ridge regression: (1) what is the structure of the estimator? (2) how to correctly use cross-validation to choose the regularization parameter? and (3) how to accelerate computation without losing too much accuracy? We consider the three problems in a unified larg…