A new method using negative-shifted gradient descent improves overparameterized linear regression by avoiding structural limitations of negative ridge endpoints.
problem Structural limitations of negative ridge endpoints in overparameterized linear regression.
method Negative-shifted gradient descent, which avoids the pole constraint of negative ridge endpoints.
result The method improves over all admissible endpoints by a polynomial factor in risk under explicit conditions.
A conventional wisdom in statistical learning is that large models require strong regularization to prevent overfitting. Here we show that this rule can be violated by linear regression in the underdetermined n≪p situation under realistic conditions. Using simulations and real-life high-dimensional data sets, we d…
Study optimal ridge regularization for out-of-distribution prediction.
problem Optimal ridge regularization for predicting out-of-distribution data.
method Established conditions for optimal regularization under covariate and regression shifts, proving monotonic risk in data aspect ratio.
result Negative regularization can be optimal under shifts, even with isotropic or underparameterized training features.
New regularization method corrects over-shrinkage in small data regression.
problem Over-shrinkage in small data regression leading to underfitting.
method Negative-capable ridge family that permits negative regularization.
result Negative regularization acts as controlled anti-shrinkage, increasing effective complexity.
New findings on how overfitting can be beneficial in ridge regression.
problem Understanding overfitting in overparameterized models.
method Extending previous results on linear regression to ridge regression, eliminating independence assumptions.
result Sharp bounds on the variance and bias terms, explaining optimal regularization in ridge regression.
The least absolute shrinkage and selection operator (lasso) and ridge regression produce usually different estimates although input, loss function and parameterization of the penalty are identical. In this paper we look for ridge and lasso models with identical solution set. It turns out, that the lasso model with shri…
Optimal SD improves ridge regression performance strictly and precisely.
problem Improving ridge regression performance through self-distillation.
method Analyzes unconstrained SD for ridge regression, deriving optimal mixing weight and asymptotic risk.
result Optimal SD strictly improves ridge regression performance, with exact risk equivalents derived.
We analyze optimal weighted ridge regression in overparameterized linear models.
problem Optimal regularization in overparameterized linear regression models.
method Generalized ridge regression with weighted regularization.
result The optimal regularization parameter can be negative in overparameterized settings.
Kernel methods identify treatment effects with unobserved confounding using negative controls.
problem Learning causal relationships with unmeasured confounding.
method Kernel ridge regression algorithms for nonparametric treatment effects.
result Uniform consistency and finite sample rates of convergence proved.
Two new ridge solutions improve BLS on added nodes, achieving better accuracy.
problem Improving the Broad Learning System (BLS) for new nodes.
method Proposed two ridge solutions for BLS output weights, updating efficiently.
result Proposed ridge solutions achieve better testing accuracy than original BLS.
Two new algorithms recover ridge lines from point clouds with convergence guarantees.
problem Extracting filamentary structure from point clouds.
method Proposes two novel algorithms with convergence guarantees.
result The algorithms can asymptotically recover the full ridge set.
Ridge regularization simplifies model complexity in data science.
problem Overfitting in statistical models.
method Adding a penalty on the magnitude of coefficients.
result Effective in reducing model complexity and improving generalization.
We study the problem of estimating the ridges of a density function. Ridge estimation is an extension of mode finding and is useful for understanding the structure of a density. It can also be used to find hidden structure in point cloud data. We show that, under mild regularity conditions, the ridges of the kernel den…
Improved ridge estimators avoid tuning parameters for high-dimensional data.
problem Difficulty in calibrating tuning parameters for ridge estimators.
method Developed modified ridge estimators that eliminate tuning parameters.
result Modified ridge estimators outperform standard methods in prediction accuracy.
New method for inference on covariates in NMF with random effects.
problem Formal inference for covariate effects in NMF with non-negativity constraints.
method NMF-RE model with random effects, ridge updates, df-based cap, asymptotic linearization, wild bootstrap.
result Valid inference on covariates with non-negativity constraint, avoiding degeneracy.
New equivalences found between subsampling and ridge regularization methods.
problem Establishing precise structural and risk equivalences between subsampling and ridge regularization.
method Proved structural and risk equivalences between subsample ridge estimators and different ridge regularization levels and subsample aspect ratios.
result Optimally tuned ridge regression exhibits a monotonic prediction risk in the data aspect ratio.
Study ridge ensembles in proportional feature-to-sample size regime, proving risk equivalence and GCV consistency.
problem Characterizing and optimizing ridge ensembles in proportional feature-to-sample size regimes.
method Proportional asymptotics analysis, GCV for tuning, proving risk equivalence.
result Risk of optimal full ridgeless ensemble matches optimal ridge predictor's risk.
We are concerned with an approximation problem for a symmetric positive semidefinite matrix due to motivation from a class of nonlinear machine learning methods. We discuss an approximation approach that we call {matrix ridge approximation}. In particular, we define the matrix ridge approximation as an incomplete matri…
Boosting ridge regression for high-dimensional data classification reduces computational cost and improves learning time.
problem High computational demand of inverting regularised covariance matrix in ridge regression for high-dimensional problems.
method Train an ensemble of ridge regressors in randomly projected subspaces, then combine them using adaptive boosting.
result Effective in terms of learning time and improved predictive performance in some cases.
Interpretable text-response modelling for structured outcomes
problem Predicting structured responses alongside textual data
method Joint non-negative matrix factorisation and binomial regression
result Recovering stable response-relevant textual signals
Short proof shows how ridge regression works with random data.
problem Understanding prediction error in ridge regression with random design.
method Combination of exchangeability arguments, matrix perturbation, and operator convexity.
result Elementary proof of prediction error without complex inequalities.
The paper examines how nonlinear transformations affect ridge sets in manifold learning.
problem Understanding the impact of nonlinear transformations on ridge sets in manifold learning.
method Examined the effects of nonlinear transformations on ridge sets using mathematical proofs and numerical experiments.
result The inclusion relationship $\cR(f\circ p)\subseteq \cR(p)$ holds for strictly increasing and concave transformations, and the Hausdorff distance between transformed and non-transformed ridge sets is smaller.
HARFE approximates sparse additive functions using random features and ridge regression.
problem Approximating high-dimensional sparse additive functions.
method Hard-ridge random feature expansion with sparse ridge regression and hard-thresholding pursuit.
result HARFE method converges with a given error bound and achieves lower error than other algorithms.
Ridge leverage scores provide a balance between low-rank approximation and regularization, and are ubiquitous in randomized linear algebra and machine learning. Deterministic algorithms are also of interest in the moderately big data regime, because deterministic algorithms provide interpretability to the practitioner …
New insights into how neural networks learn features, especially when they are very wide.
problem Understanding how gradient flow in wide neural networks selects solutions, especially in the feature-learning regime.
method Axiomatizing the canonical regularizer as a function-space energy and lift, and deriving geodesic ridge for the feature-learning regime.
result Gradient flow in feature-learning networks biases towards ridge regularization, distorting the inductive bias and damaging pretrained networks.
Unified study of ridge regression structure, cross-validation, and acceleration.
problem Understanding and optimizing ridge regression in large-data settings.
method Unified large-data linear model analysis, cross-validation bias correction, sketching accuracy study.
result Unified understanding and improved methods for ridge regression.
MGD with early stopping tends to ridge regularization in least squares regression.
problem Characterizing the implicit regularization of MGD with early stopping.
method Continuous-time view of MGD (momentum gradient flow) and comparison with explicit ridge regularization.
result Under optimal tuning, the risk of MGF is no more than 1.54 times that of ridge.
Kernel balancing weights are generalized as KRRR, providing better confidence intervals for treatment effects.
problem Lack of generalization error, correct feature specification, and limited to average effects.
method Interpreting kernel balancing weights as KRRR, relaxing feature specification, and extending Gaussian approximation.
result KRRR provides strong generalization properties and justifies confidence sets for causal functions.
Kernel ridge regression imputation with consistent variance estimation for handling missing data.
problem Handling missing data in statistical analysis.
method Kernel ridge regression imputation combined with entropy method for variance estimation.
result Root-n consistency of the imputation estimator in a Sobolev space setting.
A new method for high-dimensional functional regression reduces multicollinearity and improves interpretability.
problem Multicollinearity, overfitting, and interpretability in high-dimensional functional linear models.
method Partition-based functional ridge regression framework.
result Improved numerical stability and enhanced interpretability without explicit variable selection.
Novel algorithm identifies nonlinear Granger causal relationships using kernel ridge regression.
problem Identification of nonlinear Granger causal relationships.
method Flexible plug-in architecture with kernel ridge regression using radial basis function.
result Kernel ridge regression in mlcausality achieves competitive AUC scores and more finely calibrated p-values.
A new method corrects bias in high-dimensional ridge regression.
problem Inherent bias in ridge regression limits statistical efficiency and scalability.
method Iterative bias correction strategy for p<n and Ridge-Screening method for p>n. result Valid inferences and asymptotic properties established for de-biased ridge estimators.
Estimates modes and ridges in mixed Euclidean and directional spaces.
problem Estimating local modes and density ridges in product spaces combining Euclidean and directional metrics.
method Extends mean shift algorithm to product spaces, addressing challenges in generalization.
result Established convergence of the proposed methods and demonstrated effectiveness on real-world datasets.
Ridge regression linked to Poisson resetting in statistical physics.
problem Understanding and extending ridge regularization in machine learning.
method Connecting stochastic resetting from statistical physics with ridge regularization in machine learning, using renewal processes.
result Exact filter identities for ridge regularization in various reset laws, including exponential and non-exponential.
The study characterizes diffusion model generalization using data-dependent ridge manifolds.
problem Understanding where diffusion model-generated samples lie when not memorizing the training set.
method Introduced a time-dependent family of log-density ridge manifolds to characterize reverse-time inference.
result Generated samples evolve by a reach-align-slide mechanism, controlled by normal and tangential components of training error.
The choice of the kernel is critical to the success of many learning algorithms but it is typically left to the user. Instead, the training data can be used to learn the kernel by selecting it out of a given family, such as that of non-negative linear combinations of p base kernels, constrained by a trace or L1 regular…
Derives ideal train/test split for ridge regression in large data limit.
problem Finding optimal train/test split for ridge regression in large data scenarios.
method Mathematical derivation of optimal train/test split, considering ridge tuning parameter and asymptotic behavior.
result The optimal train/test split for ridge regression in the large data limit depends weakly on the ridge tuning parameter alpha.
Optimal ridge regularization computed iteratively from generative parameters.
problem Finding the optimal ridge regularization strength for linear regression.
method Iterative procedure to compute optimal regularization strength numerically.
result The proposed procedure attains near-optimal generalization across various conditions.
We improve prediction risk estimation for large datasets using sketching and ridge regression.
problem Estimating prediction risks for large datasets efficiently and accurately.
method Random matrix theory, generalized cross validation, sketched ridge regression ensembles, and ensemble trick.
result Consistent risk estimation and prediction intervals for large-scale datasets.
SGD implicitly regularizes linear regression problems better than ridge regression for many cases.
problem Understanding implicit regularization in linear regression problems.
method Comparing SGD and ridge regression on a broad class of least squares problems.
result SGD generalizes no worse than ridge regression for many problem instances, sometimes better.
The paper explores properties of the Radon transform in relation to neural networks and ridges.
problem Understanding the Radon transform and its application to neural networks and ridges.
method Investigates properties of the Radon transform, introduces new subspaces, and characterizes ridges for any distributional profile.
result Clarifies and simplifies results on the optimality of ReLU networks using the Radon transform.
Time-varying parameters are shown to be ridge regressions, simplifying computations and tuning.
problem Capturing structural change in economic data.
method Ridge regression approach, including cross-validation for tuning, and extensions for sparsity and reduced-rank restrictions.
result The method efficiently estimates large numbers of time-varying parameters, demonstrated with Canadian monetary policy data.
Paper proves linear convergence of SCMS algorithm for directional data.
problem Identifying density ridges in directional data.
method Generalized SCMS algorithm to directional data, derived from SCGA with adaptive step size.
result Linear convergence of the proposed directional SCMS algorithm.
Two efficient ridge solutions improve BLS for new inputs, enhancing accuracy and speed.
problem Improving BLS for new added inputs in a learning system.
method Proposes recursive and square-root BLS algorithms using inverse and inverse Cholesky factor updates.
result Both proposed ridge solutions improve BLS accuracy and speed, especially with larger lambda.
A new screening method for high-dimensional data reduces computational cost.
problem Challenges in variable selection for ultrahigh-dimensional linear regression.
method Ordering absolute sample ridge partial correlations to screen variables.
result The method provides sure screening property without strong assumptions.
Localized sketching improves matrix multiplication and ridge regression complexity.
problem Efficiently approximate matrix multiplication and ridge regression with limited data availability.
method Localized sketching matrices for block diagonal structure, reducing sample complexity.
result Localized sketching achieves sample complexity matching global sketching methods.
Ridge regression is revisited with debiasing and thresholding, offering advantages over Lasso.
problem High-dimensional data challenges classical ridge regression's sparsity detection and bias issues.
method Debiasing and thresholding ridge regression, introducing a wild bootstrap for confidence regions and hypothesis testing, and a hybrid bootstrap for prediction intervals.
result Debiased and thresholded ridge regression can offer similar performance to thresholded Lasso and may be preferable in some settings.
Improves kernel ridge regression by optimizing scale and feature parameters.
problem Kernel ridge regression with fixed kernel.
method Introduces a matrix parameter U to optimize scale and feature parameters.
result Solves a nonlinear variational problem to optimize U.