arXiv research

A conventional wisdom in statistical learning is that large models require strong regularization to prevent overfitting. Here we show that this rule can be violated by linear regression in the underdetermined $n\ll p$ situation under realistic conditions. Using simulations and real-life high-dimensional data sets, we d…

Efficient cross-validation for multi-penalty ridge regression.

We propose a penalized likelihood method to jointly estimate multiple precision matrices for use in quadratic discriminant analysis and model based clustering. A ridge penalty and a ridge fusion penalty are used to introduce shrinkage and promote similarity between precision matrix estimates. Block-wise coordinate desc…

Proposes a new ridge estimator for smooth covariates with adaptive centering.

A fast method estimates group-adaptive elastic net penalties using co-data.

Many modern statistical applications ask for the estimation of a covariance (or precision) matrix in settings where the number of variables is larger than the number of observations. There exists a broad class of ridge-type estimators that employs regularization to cope with the subsequent singularity of the sample cov…

The paper proposes a gradient-based method for multi-penalty Ridge regression.

This paper tackles the problem of selecting among several linear estimators in non-parametric regression; this includes model selection for linear regression, the choice of a regularization parameter in kernel ridge regression, spline smoothing or locally weighted regression, and the choice of a kernel in multiple kern…

ecpc R-package improves high-dimensional prediction with co-data.

Two new regularization methods improve neural network performance and complexity control.

This paper makes kernels interpretable for wide feature matrices.

Efficiently performs robust and sparse kernel regression.

Study ridge ensembles in proportional feature-to-sample size regime, proving risk equivalence and GCV consistency.

This paper considers a multiple regression model and compares, under full model hypothesis, analytically as well as by simulation, the performance characteristics of some popular penalty estimators such as ridge regression, LASSO, adaptive LASSO, SCAD, and elastic net versus Least Squares Estimator, restricted estimato…

Fast method estimates variable importance for large neural networks.

DRE combines DNN with random feature regression for efficient neural network design.

This paper presents the asymptotic behavior of a linear instrumental variables (IV) estimator that uses a ridge regression penalty. The regularization tuning parameter is selected empirically by splitting the observed data into training and test samples. Conditional on the tuning parameter, the training sample creates …

Estimates Gaussian location model with ridge regularization, comparing variational and spectral methods.

We propose the nuclear norm penalty as an alternative to the ridge penalty for regularized multinomial regression. This convex relaxation of reduced-rank multinomial regression has the advantage of leveraging underlying structure among the response categories to make better predictions. We apply our method, nuclear pen…

Ridge regression linked to Poisson resetting in statistical physics.

This paper tackles hyperparameter tuning for large-scale kernel ridge regression.

AgFlow speeds up model selection in penalized PCA.

Flexible co-data learning improves clinical prediction models.

Develops certificates for local population-risk increments using cross-fitted ridge calibration.

Enhances UPSA to reduce noise in financial data.

Training a source model optimally for its own task is suboptimal for downstream transfer.

Study on consistency of ML methods for moving objects in non-stationary environments.

The least absolute shrinkage and selection operator (lasso) and ridge regression produce usually different estimates although input, loss function and parameterization of the penalty are identical. In this paper we look for ridge and lasso models with identical solution set. It turns out, that the lasso model with shri…

When the design matrix has orthonormal columns, "soft thresholding" the ordinary least squares (OLS) solution produces the Lasso solution [Tibshirani, 1996]. If one uses the Puffer preconditioned Lasso [Jia and Rohe, 2012], then this result generalizes from orthonormal designs to full rank designs (Theorem 1). Theorem …

We consider the problem of selecting the best estimator among a family of Tikhonov regularized estimators, or, alternatively, to select a linear combination of these regularizers that is as good as the best regularizer in the family. Our theory reveals that if the Tikhonov regularizers share the same penalty matrix wit…

Ordinary least squares (OLS) is the default method for fitting linear models, but is not applicable for problems with dimensionality larger than the sample size. For these problems, we advocate the use of a generalized version of OLS motivated by ridge regression, and propose two novel three-step algorithms involving l…

This article is concerned with the Bridge Regression, which is a special family in penalized regression with penalty function $\sum_{j=1}^{p}|β_j|^q$ with $q>0$, in a linear model with linear restrictions. The proposed restricted bridge (RBRIDGE) estimator simultaneously estimates parameters and selects important varia…

There are different problems for resolution of complex LC-MS or GC-MS data, such as the existence of embedded chromatographic peaks, continuum background and overlapping in mass channels for different components. These problems cause rotational ambiguity in recovered profiles calculated using multivariate curve resolut…

The choice of normalization affects the coefficients in regularized regression models.

The paper develops a theory explaining how machine learning models can amplify biases.

Study evaluates scikit-learn regularization frameworks for machine learning models.

High-dimensional sparse modeling via regularization provides a powerful tool for analyzing large-scale data sets and obtaining meaningful, interpretable models. The use of nonconvex penalty functions shows advantage in selecting important features in high dimensions, but the global optimality of such methods still dema…

New hybrid model predicts carbon prices using blockchain data.

Improves regression efficiency by separating material and immaterial parts of responses.

Improved Kriging model reduces prediction errors.

Two new algorithms recover ridge lines from point clouds with convergence guarantees.

The original Broad Learning System (BLS) on new added nodes and its existing efficient implementation both assume the ridge parameter lambda -> 0 in the ridge inverse to approximate the generalized inverse, and compute the generalized inverse solution for the output weights. In this paper, we propose two ridge solution…

Ridge regularization simplifies model complexity in data science.

A new tradeoff between regularization and sharpness improves model performance in overparameterized settings.

We study the problem of estimating the ridges of a density function. Ridge estimation is an extension of mode finding and is useful for understanding the structure of a density. It can also be used to find hidden structure in point cloud data. We show that, under mild regularity conditions, the ridges of the kernel den…

We propose a framework for the linear prediction of a multi-way array (i.e., a tensor) from another multi-way array of arbitrary dimension, using the contracted tensor product. This framework generalizes several existing approaches, including methods to predict a scalar outcome from a tensor, a matrix from a matrix, or…

Improved ridge estimators avoid tuning parameters for high-dimensional data.

New equivalences found between subsampling and ridge regularization methods.