Two new ridge solutions improve BLS on added nodes, achieving better accuracy.
problem Improving the Broad Learning System (BLS) for new nodes.
method Proposed two ridge solutions for BLS output weights, updating efficiently.
result Proposed ridge solutions achieve better testing accuracy than original BLS.
Boosting ridge regression for high-dimensional data classification reduces computational cost and improves learning time.
problem High computational demand of inverting regularised covariance matrix in ridge regression for high-dimensional problems.
method Train an ensemble of ridge regressors in randomly projected subspaces, then combine them using adaptive boosting.
result Effective in terms of learning time and improved predictive performance in some cases.
Two efficient ridge solutions improve BLS for new inputs, enhancing accuracy and speed.
problem Improving BLS for new added inputs in a learning system.
method Proposes recursive and square-root BLS algorithms using inverse and inverse Cholesky factor updates.
result Both proposed ridge solutions improve BLS accuracy and speed, especially with larger lambda.
Study shows minimizing the norm of the ERM solution stabilizes kernel ridge-less regression.
problem Stability of kernel ridge-less regression.
method Minimizing the norm of the ERM solution to minimize CV stability.
result Interpolating solution with minimum norm minimizes CV stability.
A new method corrects bias in high-dimensional ridge regression.
problem Inherent bias in ridge regression limits statistical efficiency and scalability.
method Iterative bias correction strategy for p<n and Ridge-Screening method for p>n. result Valid inferences and asymptotic properties established for de-biased ridge estimators.
Deterministic algorithm estimates ridge regression with minimal space.
problem Estimating ridge regression solutions efficiently.
method Deterministic space-efficient algorithm using Frequent Directions.
result First o(d2) space deterministic streaming algorithm with guaranteed error. We propose LOCO, an algorithm for large-scale ridge regression which distributes the features across workers on a cluster. Important dependencies between variables are preserved using structured random projections which are cheap to compute and must only be communicated once. We show that LOCO obtains a solution which …
Proposes a new ridge estimator for smooth covariates with adaptive centering.
problem Estimating coefficients and center function for smooth covariates in linear models.
method SACR framework with convex formulation, roughness penalty, and adaptive centering.
result Improves prediction and variable selection for smooth covariates.
ASkotch solves large-scale KRR faster and better than existing methods.
problem Challenges in scaling full Kernel Ridge Regression (KRR) to large datasets.
method ASkotch: A scalable, accelerated, iterative method for full KRR.
result ASkotch provides better solutions faster than state-of-the-art solvers for full and inducing points KRR.
Ridge regression shows different behaviors in binary classification with noisy labels.
problem Binary classification with noisy labels and anisotropic cluster distributions.
method Investigation of ridge regression behavior in overparameterized settings with label noise.
result Ridge regression exhibits qualitatively different behavior based on the scale of cluster mean vectors and covariance matrices.
Novel algorithm identifies nonlinear Granger causal relationships using kernel ridge regression.
problem Identification of nonlinear Granger causal relationships.
method Flexible plug-in architecture with kernel ridge regression using radial basis function.
result Kernel ridge regression in mlcausality achieves competitive AUC scores and more finely calibrated p-values.
Kernel balancing weights are generalized as KRRR, providing better confidence intervals for treatment effects.
problem Lack of generalization error, correct feature specification, and limited to average effects.
method Interpreting kernel balancing weights as KRRR, relaxing feature specification, and extending Gaussian approximation.
result KRRR provides strong generalization properties and justifies confidence sets for causal functions.
This paper tackles hyperparameter tuning for large-scale kernel ridge regression.
problem Hyperparameter tuning is crucial but often left to users, hindering efficiency and usability.
method Proposes a complexity regularization criterion based on a data-dependent penalty for efficient optimization.
result Demonstrates the benefit of the proposed approach through extensive empirical evaluation.
Improved ridge regression with Frequent Directions for large-scale tasks.
problem Improving performance of ridge regression for large-scale data.
method Combines Frequent Directions with iterative optimization schemes.
result Achieves high accuracy in estimating bias and variance for sketched ridge regression.
New insights into how neural networks learn features, especially when they are very wide.
problem Understanding how gradient flow in wide neural networks selects solutions, especially in the feature-learning regime.
method Axiomatizing the canonical regularizer as a function-space energy and lift, and deriving geodesic ridge for the feature-learning regime.
result Gradient flow in feature-learning networks biases towards ridge regularization, distorting the inductive bias and damaging pretrained networks.
Two methods solve kernel ridge regression problems efficiently.
problem Solving kernel ridge regression problems with large datasets.
method RPCholesky and KRILL preconditioning techniques.
result Efficient solutions to KRR problems with strong guarantees.
SGD implicitly regularizes linear regression problems better than ridge regression for many cases.
problem Understanding implicit regularization in linear regression problems.
method Comparing SGD and ridge regression on a broad class of least squares problems.
result SGD generalizes no worse than ridge regression for many problem instances, sometimes better.
Ridge regularized linear models (RRLMs), such as ridge regression and the SVM, are a popular group of methods that are used in conjunction with coefficient hypothesis testing to discover explanatory variables with a significant multivariate association to a response. However, many investigators are reluctant to draw ca…
Simplifies transfer learning with deep neural networks using ridge regression.
problem High computational cost of finetuning deep models for transfer learning.
method Leverage the low-rank property of deep neural networks' feature vectors in kernel ridge regression.
result Successful on supervised and semi-supervised transfer learning tasks.
The paper improves confidence ellipsoids for ridge regression with PAC bounds.
problem Uncertainty quantification in ridge regression for insufficiently exciting inputs.
method Extension of SPS EOA algorithm to ridge regression with PAC bounds.
result Explicitly shows how regularization parameter affects region sizes and provides tighter bounds.
A new method reparameterizes ridge regression for faster, more interpretable results.
problem Challenges in selecting hyperparameter α for ridge regression.
method Fractional Ridge Regression (FRR) reparameterizes RR in terms of the ratio γ.
result FRR solutions vary with different γ, avoiding wasted calculations and manual exploration.
Improved kernel ridge regression using conjugate gradients.
problem Efficiently solving large-scale kernel ridge regression problems.
method Structured Gaussian regression model with low-rank approximation and conjugate gradients.
result Enhanced approximation of kernel ridge regressor/Gaussian process posterior mean.
The paper analyzes early stopping in linear regression and shows it's equivalent to ridge regularization.
problem Understanding the effect of early stopping on linear regression models.
method Characterization of gradient descent dynamics and analysis of excess risk.
result Early stopped solution is equivalent to minimum norm solution for a generalized ridge regularized problem.
New kernel methods estimate complex causal relationships.
problem Estimating nonparametric causal functions like dose-response curves.
method Kernel ridge regression with decomposition property.
result Uniform consistency with finite sample rates proved.
Kernel ridge regression for causal inference with missing data.
problem Estimating treatment effects with missing data in selected samples.
method Kernel ridge regression estimators for nonparametric dose response curves and semiparametric treatment effects.
result Uniform consistency and finite sample rates for continuous treatment, root-n consistency for discrete treatment.
This article proposes a novel solution for stretchy polynomial regression learning. The solution comes in primal and dual closed-forms similar to that of ridge regression. Essentially, the proposed solution stretches the covariance computation via a power term thereby compresses or amplifies the estimation. Our experim…
This paper makes kernels interpretable for wide feature matrices.
problem Loss of interpretability in kernel methods for wide feature matrices.
method Re-express kernel solution as a linear combination of original features with a special metric.
result Interpretable kernel solutions for wide feature matrices are possible.
Bayesian EM method improves ridge regression tuning without LOOCV's limitations.
problem Tuning ridge regression hyper-parameter λ efficiently and accurately. method Bayesian formulation of ridge regression with expectation maximization (EM) procedure.
result Guaranteed unique optimal solution for large n without candidate λ specification. This paper characterizes the conditional distribution properties of the finite sample ridge regression estimator and uses that result to evaluate total regression and generalization errors that incorporate the inaccuracies committed at the time of parameter estimation. The paper provides explicit formulas for those err…
We establish optimal convergence rates for a decomposition-based scalable approach to kernel ridge regression. The method is simple to describe: it randomly partitions a dataset of size N into m subsets of equal size, computes an independent kernel ridge regression estimator for each subset, then averages the local sol…
Represents neural networks as solutions to inverse problems in Banach spaces.
problem Understanding the function learned by neural networks.
method Variational framework, representer theorem, polynomial ridge splines.
result Neural networks are solutions to inverse problems in Banach spaces.
When the design matrix has orthonormal columns, "soft thresholding" the ordinary least squares (OLS) solution produces the Lasso solution [Tibshirani, 1996]. If one uses the Puffer preconditioned Lasso [Jia and Rohe, 2012], then this result generalizes from orthonormal designs to full rank designs (Theorem 1). Theorem …
Two new algorithms recover ridge lines from point clouds with convergence guarantees.
problem Extracting filamentary structure from point clouds.
method Proposes two novel algorithms with convergence guarantees.
result The algorithms can asymptotically recover the full ridge set.
The least absolute shrinkage and selection operator (lasso) and ridge regression produce usually different estimates although input, loss function and parameterization of the penalty are identical. In this paper we look for ridge and lasso models with identical solution set. It turns out, that the lasso model with shri…
Ridge regularization simplifies model complexity in data science.
problem Overfitting in statistical models.
method Adding a penalty on the magnitude of coefficients.
result Effective in reducing model complexity and improving generalization.
We study the problem of estimating the ridges of a density function. Ridge estimation is an extension of mode finding and is useful for understanding the structure of a density. It can also be used to find hidden structure in point cloud data. We show that, under mild regularity conditions, the ridges of the kernel den…
Improved ridge estimators avoid tuning parameters for high-dimensional data.
problem Difficulty in calibrating tuning parameters for ridge estimators.
method Developed modified ridge estimators that eliminate tuning parameters.
result Modified ridge estimators outperform standard methods in prediction accuracy.
New equivalences found between subsampling and ridge regularization methods.
problem Establishing precise structural and risk equivalences between subsampling and ridge regularization.
method Proved structural and risk equivalences between subsample ridge estimators and different ridge regularization levels and subsample aspect ratios.
result Optimally tuned ridge regression exhibits a monotonic prediction risk in the data aspect ratio.
Paper analyzes error bounds for learning with vector-valued RF, improving existing analyses.
problem Learning with vector-valued random features in infinite-dimensional settings.
method Direct analysis of risk functional, avoiding random matrix theory.
result Strong consistency and minimax optimal convergence rates established.
Study ridge ensembles in proportional feature-to-sample size regime, proving risk equivalence and GCV consistency.
problem Characterizing and optimizing ridge ensembles in proportional feature-to-sample size regimes.
method Proportional asymptotics analysis, GCV for tuning, proving risk equivalence.
result Risk of optimal full ridgeless ensemble matches optimal ridge predictor's risk.
A study on a surprising phase transition in model generalization error as parameters approach sample size.
problem Understanding the generalization error of overparametrized ridge models.
method Finite sample analysis using continuous Newton method and ℓ2-norm solution performance. result The generalization error decreases after the threshold p=n for ridge models. A conventional wisdom in statistical learning is that large models require strong regularization to prevent overfitting. Here we show that this rule can be violated by linear regression in the underdetermined n≪p situation under realistic conditions. Using simulations and real-life high-dimensional data sets, we d…
We are concerned with an approximation problem for a symmetric positive semidefinite matrix due to motivation from a class of nonlinear machine learning methods. We discuss an approximation approach that we call {matrix ridge approximation}. In particular, we define the matrix ridge approximation as an incomplete matri…
New findings on how overfitting can be beneficial in ridge regression.
problem Understanding overfitting in overparameterized models.
method Extending previous results on linear regression to ridge regression, eliminating independence assumptions.
result Sharp bounds on the variance and bias terms, explaining optimal regularization in ridge regression.
Short proof shows how ridge regression works with random data.
problem Understanding prediction error in ridge regression with random design.
method Combination of exchangeability arguments, matrix perturbation, and operator convexity.
result Elementary proof of prediction error without complex inequalities.
The paper examines how nonlinear transformations affect ridge sets in manifold learning.
problem Understanding the impact of nonlinear transformations on ridge sets in manifold learning.
method Examined the effects of nonlinear transformations on ridge sets using mathematical proofs and numerical experiments.
result The inclusion relationship $\cR(f\circ p)\subseteq \cR(p)$ holds for strictly increasing and concave transformations, and the Hausdorff distance between transformed and non-transformed ridge sets is smaller.
HARFE approximates sparse additive functions using random features and ridge regression.
problem Approximating high-dimensional sparse additive functions.
method Hard-ridge random feature expansion with sparse ridge regression and hard-thresholding pursuit.
result HARFE method converges with a given error bound and achieves lower error than other algorithms.
Regularized linear regression improves binary classification performance, especially with ridge and ℓ1 regularization.
problem Improving binary classification accuracy with noisy labels.
method Systematic study of regularization strengths on linear classifiers trained on noisy binary classification data.
result Ridge regression consistently improves classification error, while ℓ1 regularization can induce sparsity and ℓ∞ regularization can concentrate weights to two values.