Sharp bounds for approximating Sobolev functions by ridge functions and networks.
problem Approximating Sobolev functions with multivariate ridge functions and networks.
method Proving sharp upper and lower bounds for approximation order.
result Order of approximation asymptotically behaves as n−r/(d−ℓ). We are concerned with an approximation problem for a symmetric positive semidefinite matrix due to motivation from a class of nonlinear machine learning methods. We discuss an approximation approach that we call {matrix ridge approximation}. In particular, we define the matrix ridge approximation as an incomplete matri…
New method detects biomarker-treatment interactions in clinical trials.
problem Detecting interactions between high-dimensional biomarkers and treatments in randomized trials.
method Two-stage penalized regression screening using ridge regression for multivariate screening.
result Ridge regression screening provides greater power than traditional methods in correlated data.
Ridge regularized linear models (RRLMs), such as ridge regression and the SVM, are a popular group of methods that are used in conjunction with coefficient hypothesis testing to discover explanatory variables with a significant multivariate association to a response. However, many investigators are reluctant to draw ca…
Improves regression efficiency by separating material and immaterial parts of responses.
problem Improving estimation efficiency in nonlinear multivariate regressions.
method Kernel envelope (KENV) estimator for nonparametric response envelopes in reproducing kernel Hilbert space.
result KENV achieves lower in-sample prediction risk than kernel ridge regression in non-trivial immaterial components.
A new method for high-dimensional functional regression reduces multicollinearity and improves interpretability.
problem Multicollinearity, overfitting, and interpretability in high-dimensional functional linear models.
method Partition-based functional ridge regression framework.
result Improved numerical stability and enhanced interpretability without explicit variable selection.
Meta-learning improves predictions with generalized ridge regression in high-dimensional settings.
problem Improving meta-learning performance in high-dimensional settings.
method Generalized ridge regression applied to high-dimensional multivariate random-effects linear models.
result Optimal predictive risk achieved when using the inverse of the covariance matrix of random coefficients.
HARFE approximates sparse additive functions using random features and ridge regression.
problem Approximating high-dimensional sparse additive functions.
method Hard-ridge random feature expansion with sparse ridge regression and hard-thresholding pursuit.
result HARFE method converges with a given error bound and achieves lower error than other algorithms.
PARC uses piecewise linear predictors for regression and classification.
problem Multivariate regression and classification problems.
method Alternates between ridge and softmax regression, and cluster assignment based on accuracy and separability.
result Converges to a local minimum in a finite number of steps.
Robust multivariate linear regression methods for online and offline use.
problem Estimating parameters of multivariate Gaussian linear regression models robustly.
method Robust versions of least-square criterion with online and offline algorithms.
result Asymptotic normality of robust estimates proved under weak assumptions.
The paper explores properties of the Radon transform in relation to neural networks and ridges.
problem Understanding the Radon transform and its application to neural networks and ridges.
method Investigates properties of the Radon transform, introduces new subspaces, and characterizes ridges for any distributional profile.
result Clarifies and simplifies results on the optimality of ReLU networks using the Radon transform.
We study the problem of estimating the ridges of a density function. Ridge estimation is an extension of mode finding and is useful for understanding the structure of a density. It can also be used to find hidden structure in point cloud data. We show that, under mild regularity conditions, the ridges of the kernel den…
HAR regression improves performance on small datasets.
problem Small datasets with complex functions.
method Data-adaptive kernel ridge regression using tensor-product spline basis.
result Achieves n−1/3 convergence rate for right-continuous functions. This paper aims to solve a basic problem in distributed statistical inference: how many machines can we use in parallel computing? In kernel ridge regression, we address this question in two important settings: nonparametric estimation and hypothesis testing. Specifically, we find a range for the number of machines und…
Paper proves KRR saturation effect for smooth functions.
problem Kernel ridge regression fails to reach theoretical limits for smooth functions.
method Proof of conjectured saturation lower bound for KRR.
result Proved the conjectured saturation lower bound for KRR.
Proposes adaptive ridge regression for functional linear models with piecewise shapes.
problem Functional linear regression with unknown coefficient function.
method Adaptive piecewise function template with L2 penalization. result Improves predictive power and interpretability compared to standard methods.
Software helps teach latent variable methods in multivariate data analytics.
problem Challenges in understanding multivariate data analytics for students.
method Interactive software for comparing latent variable methods.
result Builds intuition on choosing methods and interpreting coefficients.
Kernel balancing weights are generalized as KRRR, providing better confidence intervals for treatment effects.
problem Lack of generalization error, correct feature specification, and limited to average effects.
method Interpreting kernel balancing weights as KRRR, relaxing feature specification, and extending Gaussian approximation.
result KRRR provides strong generalization properties and justifies confidence sets for causal functions.
Proposes a new ridge estimator for smooth covariates with adaptive centering.
problem Estimating coefficients and center function for smooth covariates in linear models.
method SACR framework with convex formulation, roughness penalty, and adaptive centering.
result Improves prediction and variable selection for smooth covariates.
New equivalences found between subsampling and ridge regularization methods.
problem Establishing precise structural and risk equivalences between subsampling and ridge regularization.
method Proved structural and risk equivalences between subsample ridge estimators and different ridge regularization levels and subsample aspect ratios.
result Optimally tuned ridge regression exhibits a monotonic prediction risk in the data aspect ratio.
New insights into how neural networks learn features, especially when they are very wide.
problem Understanding how gradient flow in wide neural networks selects solutions, especially in the feature-learning regime.
method Axiomatizing the canonical regularizer as a function-space energy and lift, and deriving geodesic ridge for the feature-learning regime.
result Gradient flow in feature-learning networks biases towards ridge regularization, distorting the inductive bias and damaging pretrained networks.
Study ridge ensembles in proportional feature-to-sample size regime, proving risk equivalence and GCV consistency.
problem Characterizing and optimizing ridge ensembles in proportional feature-to-sample size regimes.
method Proportional asymptotics analysis, GCV for tuning, proving risk equivalence.
result Risk of optimal full ridgeless ensemble matches optimal ridge predictor's risk.
We establish L∞ and L2 error bounds for functions of many variables that are approximated by linear combinations of ReLU (rectified linear unit) and squared ReLU ridge functions with ℓ1 and ℓ0 controls on their inner and outer parameters. With the squared ReLU ridge function, we show th…
MACE optimizes stock portfolios for maximal predictability.
problem Maximizing risk-adjusted profitability through predictable stock returns.
method Developed a machine learning algorithm (MACE) using Random Forest and Ridge Regression.
result Significant increases in predictability and profitability with minimal conditioning information.
Study predictive performance of linear regression with random functional covariates.
problem Theoretical predictive performance of linear regression with random functional covariates.
method Theoretical analysis of ridge and ridge-less least-squares regression with random functional covariates.
result Probabilistic bounds on predictive excess risk for random functional covariates.
The paper examines how nonlinear transformations affect ridge sets in manifold learning.
problem Understanding the impact of nonlinear transformations on ridge sets in manifold learning.
method Examined the effects of nonlinear transformations on ridge sets using mathematical proofs and numerical experiments.
result The inclusion relationship $\cR(f\circ p)\subseteq \cR(p)$ holds for strictly increasing and concave transformations, and the Hausdorff distance between transformed and non-transformed ridge sets is smaller.
Proposes ridge regression on Riemannian manifolds for time-series prediction.
problem Time-series prediction on Riemannian manifolds.
method Combines Riemannian least-squares fitting via Bézier curves, empirical covariance on manifolds, and Mahalanobis distance regularization.
result Significant error reduction in synthetic spherical experiments and hurricane forecasting.
Novel algorithm identifies nonlinear Granger causal relationships using kernel ridge regression.
problem Identification of nonlinear Granger causal relationships.
method Flexible plug-in architecture with kernel ridge regression using radial basis function.
result Kernel ridge regression in mlcausality achieves competitive AUC scores and more finely calibrated p-values.
The elastic energy functional of a thin elastic rod or sheet is generalized to the case of an M-dimensional manifold in N-dimensional space. We derive potentials for the stress field and curvatures and find the generalized von Karman equations for a manifold in elastic equilibrium. We perform a scaling analysis of an M…
Optimal algorithms for non-linear ridge bandits reduce burn-in cost.
problem Non-linear models introduce a burn-in period with fixed cost.
method Two-stage algorithm: find initial action, then treat locally linear.
result Two-stage algorithm is statistically optimal.
We introduce single-set spectral sparsification as a deterministic sampling based feature selection technique for regularized least squares classification, which is the classification analogue to ridge regression. The method is unsupervised and gives worst-case guarantees of the generalization power of the classificati…
We investigate the optimal structure of dynamic regression models used in multivariate time series prediction and propose a scheme to form the lagged variable structure called Backward-in-Time Selection (BTS) that takes into account feedback and multi-collinearity, often present in multivariate time series. We compare …
Improves kernel ridge regression by optimizing scale and feature parameters.
problem Kernel ridge regression with fixed kernel.
method Introduces a matrix parameter U to optimize scale and feature parameters.
result Solves a nonlinear variational problem to optimize U.
New kernel methods estimate complex causal relationships.
problem Estimating nonparametric causal functions like dose-response curves.
method Kernel ridge regression with decomposition property.
result Uniform consistency with finite sample rates proved.
We improve prediction risk estimation for large datasets using sketching and ridge regression.
problem Estimating prediction risks for large datasets efficiently and accurately.
method Random matrix theory, generalized cross validation, sketched ridge regression ensembles, and ensemble trick.
result Consistent risk estimation and prediction intervals for large-scale datasets.
Optimal CATE estimation with structured contrast functions using KRR.
problem Estimating CATEs with complex response functions in RKHS.
method Unified two-stage kernel ridge regression method for structured contrast functions.
result Minimax rates governed by contrast function complexity, enabling adaptation.
A new method for faster bandwidth selection in Gaussian kernel ridge regression.
problem Efficiently selecting the bandwidth in Gaussian kernel ridge regression.
method Formulated an approximate Jacobian expression for bandwidth selection, proposing a closed-form heuristic.
result Our method is as accurate as cross-validation and marginal likelihood maximization but up to six orders of magnitude faster.
Study the cost of overfitting in noisy KRR models.
problem Cost of overfitting in noisy kernel ridge regression.
method An agnostic view of overfitting cost as a function of sample size for any target function, using Gaussian universality ansatz and task eigenstructure.
result Characterization of benign, tempered, and catastrophic overfitting.
Paper bounds the minimal rank for kernel ridge regression approximations.
problem Efficient memory and computation for kernel ridge regression.
method Lower bound on minimal rank for reliable prediction power.
result Nyström method's computational cost is almost linear in sample size.
Unified framework for shrinkage, thresholding, and regularization in normal mean estimation and linear regression.
problem Estimation of normal mean in multivariate settings with correlated observations.
method Approximate risk minimization over a functional class of shrinkage-thresholding rules.
result Unified estimator NOMAD for shrinkage, thresholding, and regularization.
We analyze ridge interpolators in correlated factor regression models using RDT.
problem Performance analysis of ridge interpolators in correlated factor regression models.
method Utilizing Random Duality Theory (RDT), we obtain precise closed form characterizations of optimization problems.
result Ridge interpolators can smooth out the excess prediction risk and exhibit double-descent behavior.
Regularized MFPCA smooths multivariate functional data for clearer patterns.
problem Challenges in controlling roughness of multivariate functional PCs.
method ReMFPCA incorporates a roughness penalty in a penalized framework to smooth PCs.
result Smoothed multivariate functional PCs reveal clearer patterns.
Two new algorithms recover ridge lines from point clouds with convergence guarantees.
problem Extracting filamentary structure from point clouds.
method Proposes two novel algorithms with convergence guarantees.
result The algorithms can asymptotically recover the full ridge set.
The original Broad Learning System (BLS) on new added nodes and its existing efficient implementation both assume the ridge parameter lambda -> 0 in the ridge inverse to approximate the generalized inverse, and compute the generalized inverse solution for the output weights. In this paper, we propose two ridge solution…
Ridge regularization simplifies model complexity in data science.
problem Overfitting in statistical models.
method Adding a penalty on the magnitude of coefficients.
result Effective in reducing model complexity and improving generalization.
Estimation of functions of d variables is considered using ridge combinations of the form ∑k=1mc1,kφ(∑j=1dc0,j,kxj−bk) where the activation function φ is a function with bounded value and derivative. These include single-hidden layer neural networks, polynomials, …
The paper introduces new estimators for multivariate functions using Fourier methods.
problem Estimating multivariate functions like densities and regression functions.
method Monte Carlo estimators based on the Fourier integral theorem.
result Established rates of convergence for new estimators, often superior to existing methods.
Develops a method for kernel ridge regression under covariate shift using pseudo-labels.
problem Learning a regression function with small mean squared error over a target distribution with labeled data from a different feature distribution.
method Split labeled data into two subsets, conduct kernel ridge regression on each, use imputation model to fill missing labels, and select the best candidate model.
result Non-asymptotic excess risk bounds demonstrate effective adaptation to target distribution and covariate shift.