Improved ridge estimators avoid tuning parameters for high-dimensional data.
problem Difficulty in calibrating tuning parameters for ridge estimators.
method Developed modified ridge estimators that eliminate tuning parameters.
result Modified ridge estimators outperform standard methods in prediction accuracy.
We study the problem of estimating the ridges of a density function. Ridge estimation is an extension of mode finding and is useful for understanding the structure of a density. It can also be used to find hidden structure in point cloud data. We show that, under mild regularity conditions, the ridges of the kernel den…
Kernel ridge regression imputation with consistent variance estimation for handling missing data.
problem Handling missing data in statistical analysis.
method Kernel ridge regression imputation combined with entropy method for variance estimation.
result Root-n consistency of the imputation estimator in a Sobolev space setting.
Two new algorithms recover ridge lines from point clouds with convergence guarantees.
problem Extracting filamentary structure from point clouds.
method Proposes two novel algorithms with convergence guarantees.
result The algorithms can asymptotically recover the full ridge set.
The paper examines how nonlinear transformations affect ridge sets in manifold learning.
problem Understanding the impact of nonlinear transformations on ridge sets in manifold learning.
method Examined the effects of nonlinear transformations on ridge sets using mathematical proofs and numerical experiments.
result The inclusion relationship $\cR(f\circ p)\subseteq \cR(p)$ holds for strictly increasing and concave transformations, and the Hausdorff distance between transformed and non-transformed ridge sets is smaller.
Modes and ridges of the probability density function behind observed data are useful geometric features. Mode-seeking clustering assigns cluster labels by associating data samples with the nearest modes, and estimation of density ridges enables us to find lower-dimensional structures hidden in data. A key technical cha…
Estimates modes and ridges in mixed Euclidean and directional spaces.
problem Estimating local modes and density ridges in product spaces combining Euclidean and directional metrics.
method Extends mean shift algorithm to product spaces, addressing challenges in generalization.
result Established convergence of the proposed methods and demonstrated effectiveness on real-world datasets.
We improve prediction risk estimation for large datasets using sketching and ridge regression.
problem Estimating prediction risks for large datasets efficiently and accurately.
method Random matrix theory, generalized cross validation, sketched ridge regression ensembles, and ensemble trick.
result Consistent risk estimation and prediction intervals for large-scale datasets.
New equivalences found between subsampling and ridge regularization methods.
problem Establishing precise structural and risk equivalences between subsampling and ridge regularization.
method Proved structural and risk equivalences between subsample ridge estimators and different ridge regularization levels and subsample aspect ratios.
result Optimally tuned ridge regression exhibits a monotonic prediction risk in the data aspect ratio.
A new method for high-dimensional functional regression reduces multicollinearity and improves interpretability.
problem Multicollinearity, overfitting, and interpretability in high-dimensional functional linear models.
method Partition-based functional ridge regression framework.
result Improved numerical stability and enhanced interpretability without explicit variable selection.
Paper proposes methods for transfer learning with random coefficient ridge regression.
problem Estimation and prediction in high-dimensional settings with related models.
method Two estimators using weighted sums of ridge estimates from target and source models.
result Explicit expression of estimation and prediction risks derived using random matrix theory.
Gradient descent outperforms ridge regression under certain covariance matrix decay conditions.
problem Comparing the performance of gradient descent and ridge regression in linear models.
method Investigated gradient descent and ridge regression for linear regression with random isotropic ground truth.
result Gradient descent outperforms ridge regression under specific covariance matrix decay conditions.
A conventional wisdom in statistical learning is that large models require strong regularization to prevent overfitting. Here we show that this rule can be violated by linear regression in the underdetermined n≪p situation under realistic conditions. Using simulations and real-life high-dimensional data sets, we d…
We propose a penalized likelihood method to jointly estimate multiple precision matrices for use in quadratic discriminant analysis and model based clustering. A ridge penalty and a ridge fusion penalty are used to introduce shrinkage and promote similarity between precision matrix estimates. Block-wise coordinate desc…
We improve kernel ridge regression for skewed responses using oversampling and adaptive partitioning.
problem Kernel ridge regression struggles with skewed response variables, leading to poor estimates.
method Combines adaptive partitioning with oversampling to address skewed responses in kernel ridge regression.
result The proposed method yields estimates with smaller risk compared to classical methods under mild conditions.
Study on ridge regression in convolutional models shows double descent error behavior.
problem Understanding generalization and estimation error in over-parameterized convolutional models.
method Analysis of ridge estimators for convolutional linear models, derivation of exact error formulae.
result Ridge estimators exhibit double descent error behavior in high-dimensional convolutional models.
In this paper, we propose a random projection approach to estimate variance in kernel ridge regression. Our approach leads to a consistent estimator of the true variance, while being computationally more efficient. Our variance estimator is optimal for a large family of kernels, including cubic splines and Gaussian ker…
Proposes a new ridge estimator for smooth covariates with adaptive centering.
problem Estimating coefficients and center function for smooth covariates in linear models.
method SACR framework with convex formulation, roughness penalty, and adaptive centering.
result Improves prediction and variable selection for smooth covariates.
A new method corrects bias in high-dimensional ridge regression.
problem Inherent bias in ridge regression limits statistical efficiency and scalability.
method Iterative bias correction strategy for p<n and Ridge-Screening method for p>n. result Valid inferences and asymptotic properties established for de-biased ridge estimators.
Many modern statistical applications ask for the estimation of a covariance (or precision) matrix in settings where the number of variables is larger than the number of observations. There exists a broad class of ridge-type estimators that employs regularization to cope with the subsequent singularity of the sample cov…
Meta-learning improves predictions with generalized ridge regression in high-dimensional settings.
problem Improving meta-learning performance in high-dimensional settings.
method Generalized ridge regression applied to high-dimensional multivariate random-effects linear models.
result Optimal predictive risk achieved when using the inverse of the covariance matrix of random coefficients.
Deterministic algorithm estimates ridge regression with minimal space.
problem Estimating ridge regression solutions efficiently.
method Deterministic space-efficient algorithm using Frequent Directions.
result First o(d2) space deterministic streaming algorithm with guaranteed error. Improved ridge regression with Frequent Directions for large-scale tasks.
problem Improving performance of ridge regression for large-scale data.
method Combines Frequent Directions with iterative optimization schemes.
result Achieves high accuracy in estimating bias and variance for sketched ridge regression.
This paper presents the asymptotic behavior of a linear instrumental variables (IV) estimator that uses a ridge regression penalty. The regularization tuning parameter is selected empirically by splitting the observed data into training and test samples. Conditional on the tuning parameter, the training sample creates …
Extended Gauss-Markov theorem for linear estimation with bounded bias.
problem Linear estimation with bounded bias operator.
method Derive optimal estimator formulas for Nuclear and Spectral norms, analyze generalization error.
result Cross-validated Nuclear and Spectral regressors outperform Ridge regression in simulations.
Study ridge ensembles in proportional feature-to-sample size regime, proving risk equivalence and GCV consistency.
problem Characterizing and optimizing ridge ensembles in proportional feature-to-sample size regimes.
method Proportional asymptotics analysis, GCV for tuning, proving risk equivalence.
result Risk of optimal full ridgeless ensemble matches optimal ridge predictor's risk.
The paper improves confidence ellipsoids for ridge regression with PAC bounds.
problem Uncertainty quantification in ridge regression for insufficiently exciting inputs.
method Extension of SPS EOA algorithm to ridge regression with PAC bounds.
result Explicitly shows how regularization parameter affects region sizes and provides tighter bounds.
Ridge regression linked to Poisson resetting in statistical physics.
problem Understanding and extending ridge regularization in machine learning.
method Connecting stochastic resetting from statistical physics with ridge regularization in machine learning, using renewal processes.
result Exact filter identities for ridge regularization in various reset laws, including exponential and non-exponential.
Study ridge regression for non-identically distributed data with varying variances.
problem Investigate high-dimensional regression with non-identical data variance.
method Propose a random effect model and use tools from random matrix theory.
result Highlight the double descent phenomenon in high-dimensional regression for certain variance profiles.
Boosting ridge regression for high-dimensional data classification reduces computational cost and improves learning time.
problem High computational demand of inverting regularised covariance matrix in ridge regression for high-dimensional problems.
method Train an ensemble of ridge regressors in randomly projected subspaces, then combine them using adaptive boosting.
result Effective in terms of learning time and improved predictive performance in some cases.
The paper develops methods to handle missing data using regularized M-estimation in reproducing kernel Hilbert space.
problem Handling missing data in statistical analysis.
method Kernel ridge regression for imputation and maximum entropy method for propensity score estimation.
result The proposed methods achieve statistical consistency and asymptotic equivalence.
New method quantifies uncertainty in distributed regression.
problem Large datasets make traditional regression techniques ineffective.
method Data-driven approach to uncertainty quantification for averaged estimator.
result Rigorous theoretical guarantees for sup-norm consistency.
Optimal ridge regularization computed iteratively from generative parameters.
problem Finding the optimal ridge regularization strength for linear regression.
method Iterative procedure to compute optimal regularization strength numerically.
result The proposed procedure attains near-optimal generalization across various conditions.
We introduce the concept of coverage risk as an error measure for density ridge estimation. The coverage risk generalizes the mean integrated square error to set estimation. We propose two risk estimators for the coverage risk and we show that we can select tuning parameters by minimizing the estimated risk. We study t…
Proposes an efficient shrinkage path for ridge regression.
problem Ill-conditioned data in linear models.
method A new generalized ridge regression shrinkage path that minimizes MSE risk.
result The path is as short as possible while maintaining optimal trade-off.
New kernel methods estimate complex causal relationships.
problem Estimating nonparametric causal functions like dose-response curves.
method Kernel ridge regression with decomposition property.
result Uniform consistency with finite sample rates proved.
Paper proves linear convergence of SCMS algorithm for directional data.
problem Identifying density ridges in directional data.
method Generalized SCMS algorithm to directional data, derived from SCGA with adaptive step size.
result Linear convergence of the proposed directional SCMS algorithm.
New insights into ridge regression with correlated data, improving risk prediction.
problem Understanding and predicting risk in ridge regression with correlated samples.
method Random matrix theory and free probability for asymptotic analysis; modified GCV estimator (CorrGCV) for unbiased prediction.
result GCV estimator fails for out-of-sample risk with correlated data; CorrGCV provides an unbiased estimator.
A new screening method for high-dimensional data reduces computational cost.
problem Challenges in variable selection for ultrahigh-dimensional linear regression.
method Ordering absolute sample ridge partial correlations to screen variables.
result The method provides sure screening property without strong assumptions.
Develops a method for kernel ridge regression under covariate shift using pseudo-labels.
problem Learning a regression function with small mean squared error over a target distribution with labeled data from a different feature distribution.
method Split labeled data into two subsets, conduct kernel ridge regression on each, use imputation model to fill missing labels, and select the best candidate model.
result Non-asymptotic excess risk bounds demonstrate effective adaptation to target distribution and covariate shift.
Time-varying parameters are shown to be ridge regressions, simplifying computations and tuning.
problem Capturing structural change in economic data.
method Ridge regression approach, including cross-validation for tuning, and extensions for sparsity and reduced-rank restrictions.
result The method efficiently estimates large numbers of time-varying parameters, demonstrated with Canadian monetary policy data.
We establish optimal convergence rates for a decomposition-based scalable approach to kernel ridge regression. The method is simple to describe: it randomly partitions a dataset of size N into m subsets of equal size, computes an independent kernel ridge regression estimator for each subset, then averages the local sol…
Kernel ridge regression inference for nonstandard data.
problem Inferential theory for kernel ridge regression with nonstandard data.
method Constructs valid and sharp confidence sets using anti-symmetric multipliers.
result Develops a test for match effects in school matching mechanisms.
Estimates Gaussian location model with ridge regularization, comparing variational and spectral methods.
problem Estimating parameters in Gaussian location model with regularization.
method Ridge-regularized log-density-ratio estimation, variational and spectral approaches.
result Regularized variational estimator has lower risk with many observations, spectral estimator with fewer observations.
ParK efficiently solves kernel ridge regression for large datasets.
problem Large-scale kernel ridge regression efficiency and accuracy.
method Partitioning feature space with random projections and iterative optimization.
result Provably maintains statistical accuracy with reduced space and time complexity.
We study the following three fundamental problems about ridge regression: (1) what is the structure of the estimator? (2) how to correctly use cross-validation to choose the regularization parameter? and (3) how to accelerate computation without losing too much accuracy? We consider the three problems in a unified larg…
Optimal CATE estimation with structured contrast functions using KRR.
problem Estimating CATEs with complex response functions in RKHS.
method Unified two-stage kernel ridge regression method for structured contrast functions.
result Minimax rates governed by contrast function complexity, enabling adaptation.
Study shows how feature weighting affects neural network regularization.
problem Understanding how feature weighting influences neural network regularization.
method Derived equivalence paths connecting different weighting matrices and ridge regularization levels.
result Ridge estimators trained on weighted features are asymptotically equivalent when evaluated against test vectors.