Boosting ridge regression for high-dimensional data classification reduces computational cost and improves learning time.
problem High computational demand of inverting regularised covariance matrix in ridge regression for high-dimensional problems.
method Train an ensemble of ridge regressors in randomly projected subspaces, then combine them using adaptive boosting.
result Effective in terms of learning time and improved predictive performance in some cases.
Short proof shows how ridge regression works with random data.
problem Understanding prediction error in ridge regression with random design.
method Combination of exchangeability arguments, matrix perturbation, and operator convexity.
result Elementary proof of prediction error without complex inequalities.
Kernel ridge regression imputation with consistent variance estimation for handling missing data.
problem Handling missing data in statistical analysis.
method Kernel ridge regression imputation combined with entropy method for variance estimation.
result Root-n consistency of the imputation estimator in a Sobolev space setting.
Prevalidated ridge regression simplifies logistic regression for high-dimensional data.
problem Efficient probabilistic classification in high-dimensional data with logistic regression.
method Developed a prevalidated ridge regression model that matches logistic regression's performance but is more computationally efficient.
result Prevalidated ridge regression achieves similar classification error and log-loss to logistic regression for high-dimensional data.
Derives ideal train/test split for ridge regression in large data limit.
problem Finding optimal train/test split for ridge regression in large data scenarios.
method Mathematical derivation of optimal train/test split, considering ridge tuning parameter and asymptotic behavior.
result The optimal train/test split for ridge regression in the large data limit depends weakly on the ridge tuning parameter alpha.
Novel algorithm identifies nonlinear Granger causal relationships using kernel ridge regression.
problem Identification of nonlinear Granger causal relationships.
method Flexible plug-in architecture with kernel ridge regression using radial basis function.
result Kernel ridge regression in mlcausality achieves competitive AUC scores and more finely calibrated p-values.
GD outperforms ridge regression and SGD in linear regression problems.
problem Comparing the risks of GD, ridge regression, and SGD in linear regression problems.
method Instance-wise finite-sample risk analysis of GD, ridge regression, and SGD.
result GD outperforms ridge regression and is incomparable with SGD in some cases.
Paper proposes methods for transfer learning with random coefficient ridge regression.
problem Estimation and prediction in high-dimensional settings with related models.
method Two estimators using weighted sums of ridge estimates from target and source models.
result Explicit expression of estimation and prediction risks derived using random matrix theory.
SGD implicitly regularizes linear regression problems better than ridge regression for many cases.
problem Understanding implicit regularization in linear regression problems.
method Comparing SGD and ridge regression on a broad class of least squares problems.
result SGD generalizes no worse than ridge regression for many problem instances, sometimes better.
Proposes ridge regression on Riemannian manifolds for time-series prediction.
problem Time-series prediction on Riemannian manifolds.
method Combines Riemannian least-squares fitting via Bézier curves, empirical covariance on manifolds, and Mahalanobis distance regularization.
result Significant error reduction in synthetic spherical experiments and hurricane forecasting.
Gradient descent outperforms ridge regression under certain covariance matrix decay conditions.
problem Comparing the performance of gradient descent and ridge regression in linear models.
method Investigated gradient descent and ridge regression for linear regression with random isotropic ground truth.
result Gradient descent outperforms ridge regression under specific covariance matrix decay conditions.
Ridge regression is revisited with debiasing and thresholding, offering advantages over Lasso.
problem High-dimensional data challenges classical ridge regression's sparsity detection and bias issues.
method Debiasing and thresholding ridge regression, introducing a wild bootstrap for confidence regions and hypothesis testing, and a hybrid bootstrap for prediction intervals.
result Debiased and thresholded ridge regression can offer similar performance to thresholded Lasso and may be preferable in some settings.
Kernel ridge regression inference for nonstandard data.
problem Inferential theory for kernel ridge regression with nonstandard data.
method Constructs valid and sharp confidence sets using anti-symmetric multipliers.
result Develops a test for match effects in school matching mechanisms.
pGMM kernel outperforms ordinary ridge regression and RBF kernel ridge regression without tuning.
problem Comparing pGMM kernel regression with other ridge regression methods.
method Implemented and compared pGMM kernel regression with ordinary ridge regression and RBF kernel ridge regression.
result pGMM kernel performs well without tuning and can match boosted trees with parameter tuning.
Ridge leverage scores provide a balance between low-rank approximation and regularization, and are ubiquitous in randomized linear algebra and machine learning. Deterministic algorithms are also of interest in the moderately big data regime, because deterministic algorithms provide interpretability to the practitioner …
We compare the risk of ridge regression to a simple variant of ordinary least squares, in which one simply projects the data onto a finite dimensional subspace (as specified by a Principal Component Analysis) and then performs an ordinary (un-regularized) least squares regression in this subspace. This note shows that …
Localized sketching improves matrix multiplication and ridge regression complexity.
problem Efficiently approximate matrix multiplication and ridge regression with limited data availability.
method Localized sketching matrices for block diagonal structure, reducing sample complexity.
result Localized sketching achieves sample complexity matching global sketching methods.
OKRidge solves sparse ridge regression problems for nonlinear systems.
problem Identifying sparse governing equations for nonlinear dynamical systems.
method OKRidge algorithm using saddle point formulation and ADMM-based approach with efficient proximal operators.
result OKRidge achieves provable optimality with significantly faster run times than Gurobi.
HARFE approximates sparse additive functions using random features and ridge regression.
problem Approximating high-dimensional sparse additive functions.
method Hard-ridge random feature expansion with sparse ridge regression and hard-thresholding pursuit.
result HARFE method converges with a given error bound and achieves lower error than other algorithms.
Improved ridge regression with Frequent Directions for large-scale tasks.
problem Improving performance of ridge regression for large-scale data.
method Combines Frequent Directions with iterative optimization schemes.
result Achieves high accuracy in estimating bias and variance for sketched ridge regression.
Improves kernel ridge regression by optimizing scale and feature parameters.
problem Kernel ridge regression with fixed kernel.
method Introduces a matrix parameter U to optimize scale and feature parameters.
result Solves a nonlinear variational problem to optimize U.
Time-varying parameters are shown to be ridge regressions, simplifying computations and tuning.
problem Capturing structural change in economic data.
method Ridge regression approach, including cross-validation for tuning, and extensions for sparsity and reduced-rank restrictions.
result The method efficiently estimates large numbers of time-varying parameters, demonstrated with Canadian monetary policy data.
A new method for high-dimensional functional regression reduces multicollinearity and improves interpretability.
problem Multicollinearity, overfitting, and interpretability in high-dimensional functional linear models.
method Partition-based functional ridge regression framework.
result Improved numerical stability and enhanced interpretability without explicit variable selection.
Proposes an efficient shrinkage path for ridge regression.
problem Ill-conditioned data in linear models.
method A new generalized ridge regression shrinkage path that minimizes MSE risk.
result The path is as short as possible while maintaining optimal trade-off.
MGD with early stopping tends to ridge regularization in least squares regression.
problem Characterizing the implicit regularization of MGD with early stopping.
method Continuous-time view of MGD (momentum gradient flow) and comparison with explicit ridge regularization.
result Under optimal tuning, the risk of MGF is no more than 1.54 times that of ridge.
Paper proves convergence rates for Gaussian kernel ridge regression.
problem Understanding convergence rates for Gaussian kernel ridge regression.
method Establishes polynomial convergence rates for KRR with fixed hyperparameters.
result First polynomial convergence rates for Gaussian kernel ridge regression.
A new method corrects bias in high-dimensional ridge regression.
problem Inherent bias in ridge regression limits statistical efficiency and scalability.
method Iterative bias correction strategy for p<n and Ridge-Screening method for p>n. result Valid inferences and asymptotic properties established for de-biased ridge estimators.
Study ridge regression for non-identically distributed data with varying variances.
problem Investigate high-dimensional regression with non-identical data variance.
method Propose a random effect model and use tools from random matrix theory.
result Highlight the double descent phenomenon in high-dimensional regression for certain variance profiles.
HAR regression improves performance on small datasets.
problem Small datasets with complex functions.
method Data-adaptive kernel ridge regression using tensor-product spline basis.
result Achieves n−1/3 convergence rate for right-continuous functions. Researchers compare different gradient methods for ridge regression, finding conjugate gradients have similar performance.
problem Comparing statistical properties of different gradient methods in ridge regression.
method Explicit non-standard error decomposition to bound prediction error of conjugate gradient iterates.
result Conjugate gradient iterates share optimality properties with gradient flow and ridge regression up to a constant factor.
DRE combines DNN with random feature regression for efficient neural network design.
problem Designing and training deep neural networks (DNN) efficiently and effectively.
method DRE architecture with two-layer neural networks, randomly drawn input and output weights trained with linear ridge regression.
result DRE outperforms state-of-the-art DNN in many data sets with lower computational cost.
Ridge regression shows different behaviors in binary classification with noisy labels.
problem Binary classification with noisy labels and anisotropic cluster distributions.
method Investigation of ridge regression behavior in overparameterized settings with label noise.
result Ridge regression exhibits qualitatively different behavior based on the scale of cluster mean vectors and covariance matrices.
A new method reparameterizes ridge regression for faster, more interpretable results.
problem Challenges in selecting hyperparameter α for ridge regression.
method Fractional Ridge Regression (FRR) reparameterizes RR in terms of the ratio γ.
result FRR solutions vary with different γ, avoiding wasted calculations and manual exploration.
ParK efficiently solves kernel ridge regression for large datasets.
problem Large-scale kernel ridge regression efficiency and accuracy.
method Partitioning feature space with random projections and iterative optimization.
result Provably maintains statistical accuracy with reduced space and time complexity.
We study the following three fundamental problems about ridge regression: (1) what is the structure of the estimator? (2) how to correctly use cross-validation to choose the regularization parameter? and (3) how to accelerate computation without losing too much accuracy? We consider the three problems in a unified larg…
Optimal ridge regularization computed iteratively from generative parameters.
problem Finding the optimal ridge regularization strength for linear regression.
method Iterative procedure to compute optimal regularization strength numerically.
result The proposed procedure attains near-optimal generalization across various conditions.
We improve kernel ridge regression for skewed responses using oversampling and adaptive partitioning.
problem Kernel ridge regression struggles with skewed response variables, leading to poor estimates.
method Combines adaptive partitioning with oversampling to address skewed responses in kernel ridge regression.
result The proposed method yields estimates with smaller risk compared to classical methods under mild conditions.
Develops a method for kernel ridge regression under covariate shift using pseudo-labels.
problem Learning a regression function with small mean squared error over a target distribution with labeled data from a different feature distribution.
method Split labeled data into two subsets, conduct kernel ridge regression on each, use imputation model to fill missing labels, and select the best candidate model.
result Non-asymptotic excess risk bounds demonstrate effective adaptation to target distribution and covariate shift.
The paper analyzes and improves the learning rates of distributed kernel ridge regression.
problem Generalization performance and learning rates of distributed kernel ridge regression.
method The paper derives optimal learning rates for DKRR in expectation and probability, proposes a communication strategy to improve learning performance, and evaluates these through theory and experiments.
result The communication strategy significantly improves the learning performance of DKRR, as demonstrated by both theoretical assessments and numerical experiments.
New method quantifies uncertainty in distributed regression.
problem Large datasets make traditional regression techniques ineffective.
method Data-driven approach to uncertainty quantification for averaged estimator.
result Rigorous theoretical guarantees for sup-norm consistency.
We improve prediction risk estimation for large datasets using sketching and ridge regression.
problem Estimating prediction risks for large datasets efficiently and accurately.
method Random matrix theory, generalized cross validation, sketched ridge regression ensembles, and ensemble trick.
result Consistent risk estimation and prediction intervals for large-scale datasets.
Meta-learning improves predictions with generalized ridge regression in high-dimensional settings.
problem Improving meta-learning performance in high-dimensional settings.
method Generalized ridge regression applied to high-dimensional multivariate random-effects linear models.
result Optimal predictive risk achieved when using the inverse of the covariance matrix of random coefficients.
The paper improves confidence ellipsoids for ridge regression with PAC bounds.
problem Uncertainty quantification in ridge regression for insufficiently exciting inputs.
method Extension of SPS EOA algorithm to ridge regression with PAC bounds.
result Explicitly shows how regularization parameter affects region sizes and provides tighter bounds.
Paper proves KRR saturation effect for smooth functions.
problem Kernel ridge regression fails to reach theoretical limits for smooth functions.
method Proof of conjectured saturation lower bound for KRR.
result Proved the conjectured saturation lower bound for KRR.
Efficient cross-validation for multi-penalty ridge regression.
problem Optimizing penalties in cross-validation for multi-penalty ridge regression.
method Developed a computationally efficient formula for the multi-penalty, sample-weighted hat-matrix.
result Significant speed-up in cross-validation for ridge regression models.
We propose LOCO, an algorithm for large-scale ridge regression which distributes the features across workers on a cluster. Important dependencies between variables are preserved using structured random projections which are cheap to compute and must only be communicated once. We show that LOCO obtains a solution which …
Bayes-optimal learning of deep random networks with Gaussian weights is studied.
problem Learning a target function corresponding to a deep, extensive-width, non-linear neural network with random Gaussian weights.
method Closed-form expressions for Bayes-optimal test error, ridge regression, kernel and random features regression are computed.
result Optimally regularized ridge regression and kernel regression achieve Bayes-optimal performances, while logistic loss yields a near-optimal test error for classification.
This paper focuses on learning rate analysis of distributed kernel ridge regression for strong mixing sequences. Using a recently developed integral operator approach and a classical covariance inequality for Banach-valued strong mixing sequences, we succeed in deriving optimal learning rate for distributed kernel ridg…