A new method for high-dimensional functional regression reduces multicollinearity and improves interpretability.
problem Multicollinearity, overfitting, and interpretability in high-dimensional functional linear models.
method Partition-based functional ridge regression framework.
result Improved numerical stability and enhanced interpretability without explicit variable selection.
HARFE approximates sparse additive functions using random features and ridge regression.
problem Approximating high-dimensional sparse additive functions.
method Hard-ridge random feature expansion with sparse ridge regression and hard-thresholding pursuit.
result HARFE method converges with a given error bound and achieves lower error than other algorithms.
HAR regression improves performance on small datasets.
problem Small datasets with complex functions.
method Data-adaptive kernel ridge regression using tensor-product spline basis.
result Achieves n−1/3 convergence rate for right-continuous functions. Paper proves KRR saturation effect for smooth functions.
problem Kernel ridge regression fails to reach theoretical limits for smooth functions.
method Proof of conjectured saturation lower bound for KRR.
result Proved the conjectured saturation lower bound for KRR.
Proposes adaptive ridge regression for functional linear models with piecewise shapes.
problem Functional linear regression with unknown coefficient function.
method Adaptive piecewise function template with L2 penalization. result Improves predictive power and interpretability compared to standard methods.
Study predictive performance of linear regression with random functional covariates.
problem Theoretical predictive performance of linear regression with random functional covariates.
method Theoretical analysis of ridge and ridge-less least-squares regression with random functional covariates.
result Probabilistic bounds on predictive excess risk for random functional covariates.
Proposes ridge regression on Riemannian manifolds for time-series prediction.
problem Time-series prediction on Riemannian manifolds.
method Combines Riemannian least-squares fitting via Bézier curves, empirical covariance on manifolds, and Mahalanobis distance regularization.
result Significant error reduction in synthetic spherical experiments and hurricane forecasting.
Novel algorithm identifies nonlinear Granger causal relationships using kernel ridge regression.
problem Identification of nonlinear Granger causal relationships.
method Flexible plug-in architecture with kernel ridge regression using radial basis function.
result Kernel ridge regression in mlcausality achieves competitive AUC scores and more finely calibrated p-values.
Improves kernel ridge regression by optimizing scale and feature parameters.
problem Kernel ridge regression with fixed kernel.
method Introduces a matrix parameter U to optimize scale and feature parameters.
result Solves a nonlinear variational problem to optimize U.
We introduce single-set spectral sparsification as a deterministic sampling based feature selection technique for regularized least squares classification, which is the classification analogue to ridge regression. The method is unsupervised and gives worst-case guarantees of the generalization power of the classificati…
Kernel balancing weights are generalized as KRRR, providing better confidence intervals for treatment effects.
problem Lack of generalization error, correct feature specification, and limited to average effects.
method Interpreting kernel balancing weights as KRRR, relaxing feature specification, and extending Gaussian approximation.
result KRRR provides strong generalization properties and justifies confidence sets for causal functions.
A new method for faster bandwidth selection in Gaussian kernel ridge regression.
problem Efficiently selecting the bandwidth in Gaussian kernel ridge regression.
method Formulated an approximate Jacobian expression for bandwidth selection, proposing a closed-form heuristic.
result Our method is as accurate as cross-validation and marginal likelihood maximization but up to six orders of magnitude faster.
New kernel methods estimate complex causal relationships.
problem Estimating nonparametric causal functions like dose-response curves.
method Kernel ridge regression with decomposition property.
result Uniform consistency with finite sample rates proved.
Develops a method for kernel ridge regression under covariate shift using pseudo-labels.
problem Learning a regression function with small mean squared error over a target distribution with labeled data from a different feature distribution.
method Split labeled data into two subsets, conduct kernel ridge regression on each, use imputation model to fill missing labels, and select the best candidate model.
result Non-asymptotic excess risk bounds demonstrate effective adaptation to target distribution and covariate shift.
Bayes-optimal learning of deep random networks with Gaussian weights is studied.
problem Learning a target function corresponding to a deep, extensive-width, non-linear neural network with random Gaussian weights.
method Closed-form expressions for Bayes-optimal test error, ridge regression, kernel and random features regression are computed.
result Optimally regularized ridge regression and kernel regression achieve Bayes-optimal performances, while logistic loss yields a near-optimal test error for classification.
We improve prediction risk estimation for large datasets using sketching and ridge regression.
problem Estimating prediction risks for large datasets efficiently and accurately.
method Random matrix theory, generalized cross validation, sketched ridge regression ensembles, and ensemble trick.
result Consistent risk estimation and prediction intervals for large-scale datasets.
Paper bounds the minimal rank for kernel ridge regression approximations.
problem Efficient memory and computation for kernel ridge regression.
method Lower bound on minimal rank for reliable prediction power.
result Nyström method's computational cost is almost linear in sample size.
Optimal rates for vector-valued regression on various norms.
problem Optimal rates for vector-valued ridge regression on continuous norms.
method Combining standard capacity assumptions with tensor product constructions of vector-valued interpolation spaces.
result Optimal rates for vector-valued ridge regression, independent of output space dimension.
Optimal CATE estimation with structured contrast functions using KRR.
problem Estimating CATEs with complex response functions in RKHS.
method Unified two-stage kernel ridge regression method for structured contrast functions.
result Minimax rates governed by contrast function complexity, enabling adaptation.
Boosting ridge regression for high-dimensional data classification reduces computational cost and improves learning time.
problem High computational demand of inverting regularised covariance matrix in ridge regression for high-dimensional problems.
method Train an ensemble of ridge regressors in randomly projected subspaces, then combine them using adaptive boosting.
result Effective in terms of learning time and improved predictive performance in some cases.
We analyze ridge interpolators in correlated factor regression models using RDT.
problem Performance analysis of ridge interpolators in correlated factor regression models.
method Utilizing Random Duality Theory (RDT), we obtain precise closed form characterizations of optimization problems.
result Ridge interpolators can smooth out the excess prediction risk and exhibit double-descent behavior.
Improved kernel ridge regression for large datasets using weighted random binning.
problem Efficiently approximating kernel matrices for large-scale datasets.
method Introduced weighted random binning features for locality sensitive hashing.
result Weighted random binning features generate Gaussian processes of any desired smoothness.
New equivalences found between subsampling and ridge regularization methods.
problem Establishing precise structural and risk equivalences between subsampling and ridge regularization.
method Proved structural and risk equivalences between subsample ridge estimators and different ridge regularization levels and subsample aspect ratios.
result Optimally tuned ridge regression exhibits a monotonic prediction risk in the data aspect ratio.
Study confirms learning rates for vector-valued spectral algorithms, proving consistency.
problem Theoretical confirmation of learning rates for vector-valued spectral algorithms.
method Rigorous analysis of learning rates for various vector-valued spectral algorithms, including kernel ridge regression and gradient descent.
result Upper and lower bounds on learning rates for vector-valued spectral algorithms, proving minimax optimality in various scenarios.
Short proof shows how ridge regression works with random data.
problem Understanding prediction error in ridge regression with random design.
method Combination of exchangeability arguments, matrix perturbation, and operator convexity.
result Elementary proof of prediction error without complex inequalities.
Study the cost of overfitting in noisy KRR models.
problem Cost of overfitting in noisy kernel ridge regression.
method An agnostic view of overfitting cost as a function of sample size for any target function, using Gaussian universality ansatz and task eigenstructure.
result Characterization of benign, tempered, and catastrophic overfitting.
Kernel ridge regression imputation with consistent variance estimation for handling missing data.
problem Handling missing data in statistical analysis.
method Kernel ridge regression imputation combined with entropy method for variance estimation.
result Root-n consistency of the imputation estimator in a Sobolev space setting.
Ridge regression analysis under varying sample size and dimensionality.
problem Prediction error analysis in asymptotic ridge regression.
method Characterization of prediction error based on covariance and parameter structure.
result Interpolation can be optimal even with bounded SNR if true parameter coefficients are larger on high-variance directions.
Prevalidated ridge regression simplifies logistic regression for high-dimensional data.
problem Efficient probabilistic classification in high-dimensional data with logistic regression.
method Developed a prevalidated ridge regression model that matches logistic regression's performance but is more computationally efficient.
result Prevalidated ridge regression achieves similar classification error and log-loss to logistic regression for high-dimensional data.
PCHAL and PCHAR use principal components to speed up HAL and HAR methods.
problem Computational infeasibility in high dimensions for HAL and HAR.
method Outcome-blind principal component reduction of HAL basis.
result Empirical performance comparable to HAL and HAR, with computational gains.
Derives ideal train/test split for ridge regression in large data limit.
problem Finding optimal train/test split for ridge regression in large data scenarios.
method Mathematical derivation of optimal train/test split, considering ridge tuning parameter and asymptotic behavior.
result The optimal train/test split for ridge regression in the large data limit depends weakly on the ridge tuning parameter alpha.
Paper introduces robust kernel ridge regression using Cauchy loss for handling various noise types.
problem Developing robust regression methods for noisy data.
method Introduces kernel Cauchy ridge regressor (KCRR) using Cauchy loss function.
result Establishes almost minimax-optimal convergence rate for KCRR in terms of L2-risk. GD outperforms ridge regression and SGD in linear regression problems.
problem Comparing the risks of GD, ridge regression, and SGD in linear regression problems.
method Instance-wise finite-sample risk analysis of GD, ridge regression, and SGD.
result GD outperforms ridge regression and is incomparable with SGD in some cases.
Paper proposes methods for transfer learning with random coefficient ridge regression.
problem Estimation and prediction in high-dimensional settings with related models.
method Two estimators using weighted sums of ridge estimates from target and source models.
result Explicit expression of estimation and prediction risks derived using random matrix theory.
SGD implicitly regularizes linear regression problems better than ridge regression for many cases.
problem Understanding implicit regularization in linear regression problems.
method Comparing SGD and ridge regression on a broad class of least squares problems.
result SGD generalizes no worse than ridge regression for many problem instances, sometimes better.
Gradient descent outperforms ridge regression under certain covariance matrix decay conditions.
problem Comparing the performance of gradient descent and ridge regression in linear models.
method Investigated gradient descent and ridge regression for linear regression with random isotropic ground truth.
result Gradient descent outperforms ridge regression under specific covariance matrix decay conditions.
Ridge regression is revisited with debiasing and thresholding, offering advantages over Lasso.
problem High-dimensional data challenges classical ridge regression's sparsity detection and bias issues.
method Debiasing and thresholding ridge regression, introducing a wild bootstrap for confidence regions and hypothesis testing, and a hybrid bootstrap for prediction intervals.
result Debiased and thresholded ridge regression can offer similar performance to thresholded Lasso and may be preferable in some settings.
Kernel ridge regression inference for nonstandard data.
problem Inferential theory for kernel ridge regression with nonstandard data.
method Constructs valid and sharp confidence sets using anti-symmetric multipliers.
result Develops a test for match effects in school matching mechanisms.
pGMM kernel outperforms ordinary ridge regression and RBF kernel ridge regression without tuning.
problem Comparing pGMM kernel regression with other ridge regression methods.
method Implemented and compared pGMM kernel regression with ordinary ridge regression and RBF kernel ridge regression.
result pGMM kernel performs well without tuning and can match boosted trees with parameter tuning.
Ridge leverage scores provide a balance between low-rank approximation and regularization, and are ubiquitous in randomized linear algebra and machine learning. Deterministic algorithms are also of interest in the moderately big data regime, because deterministic algorithms provide interpretability to the practitioner …
We compare the risk of ridge regression to a simple variant of ordinary least squares, in which one simply projects the data onto a finite dimensional subspace (as specified by a Principal Component Analysis) and then performs an ordinary (un-regularized) least squares regression in this subspace. This note shows that …
Localized sketching improves matrix multiplication and ridge regression complexity.
problem Efficiently approximate matrix multiplication and ridge regression with limited data availability.
method Localized sketching matrices for block diagonal structure, reducing sample complexity.
result Localized sketching achieves sample complexity matching global sketching methods.
OKRidge solves sparse ridge regression problems for nonlinear systems.
problem Identifying sparse governing equations for nonlinear dynamical systems.
method OKRidge algorithm using saddle point formulation and ADMM-based approach with efficient proximal operators.
result OKRidge achieves provable optimality with significantly faster run times than Gurobi.
Improved ridge regression with Frequent Directions for large-scale tasks.
problem Improving performance of ridge regression for large-scale data.
method Combines Frequent Directions with iterative optimization schemes.
result Achieves high accuracy in estimating bias and variance for sketched ridge regression.
New estimator for estimating derivatives in nonparametric regression.
problem Estimating derivatives of regression functions.
method Plug-in kernel ridge regression (KRR) estimator.
result Plug-in property for derivatives estimation, optimal rate of convergence.
Paper analyzes distributed learning with non-i.i.d. samples.
problem Learning rate analysis for distributed kernel ridge regression with dependent samples.
method Integral operator approach and covariance inequality for strong mixing sequences.
result Derives optimal learning rates for distributed kernel ridge regression.
Time-varying parameters are shown to be ridge regressions, simplifying computations and tuning.
problem Capturing structural change in economic data.
method Ridge regression approach, including cross-validation for tuning, and extensions for sparsity and reduced-rank restrictions.
result The method efficiently estimates large numbers of time-varying parameters, demonstrated with Canadian monetary policy data.
Proposes an efficient shrinkage path for ridge regression.
problem Ill-conditioned data in linear models.
method A new generalized ridge regression shrinkage path that minimizes MSE risk.
result The path is as short as possible while maintaining optimal trade-off.