New equivalences found between subsampling and ridge regularization methods.
problem Establishing precise structural and risk equivalences between subsampling and ridge regularization.
method Proved structural and risk equivalences between subsample ridge estimators and different ridge regularization levels and subsample aspect ratios.
result Optimally tuned ridge regression exhibits a monotonic prediction risk in the data aspect ratio.
Ridge regularization simplifies model complexity in data science.
problem Overfitting in statistical models.
method Adding a penalty on the magnitude of coefficients.
result Effective in reducing model complexity and improving generalization.
Interpolation hurts robust generalization even without noise.
problem The challenge of robust generalization in the absence of noise.
method Avoiding interpolation through ridge regularization.
result Ridge regularization improves robust generalization.
Study optimal ridge regularization for out-of-distribution prediction.
problem Optimal ridge regularization for predicting out-of-distribution data.
method Established conditions for optimal regularization under covariate and regression shifts, proving monotonic risk in data aspect ratio.
result Negative regularization can be optimal under shifts, even with isotropic or underparameterized training features.
Unified view connects CoCoA and ADMM for distributed ERM.
problem Connection between CoCoA and ADMM for distributed ERM.
method Unified primal-dual perspective reformulation.
result Unified ADMM variants perform at least as good as CoCoA in ridge-regularized ERM.
The paper analyzes a simple neural network model with algebraic methods.
problem Finding minima of a ridge-regularized mean squared error for ReLU perceptrons.
method Developed a Divide-Enumerate-Merge strategy using computational algebra.
result Identifies both isolated and connected minima of the RR-MSE.
The paper analyzes early stopping in linear regression and shows it's equivalent to ridge regularization.
problem Understanding the effect of early stopping on linear regression models.
method Characterization of gradient descent dynamics and analysis of excess risk.
result Early stopped solution is equivalent to minimum norm solution for a generalized ridge regularized problem.
Estimates Gaussian location model with ridge regularization, comparing variational and spectral methods.
problem Estimating parameters in Gaussian location model with regularization.
method Ridge-regularized log-density-ratio estimation, variational and spectral approaches.
result Regularized variational estimator has lower risk with many observations, spectral estimator with fewer observations.
Study high-dimensional logistic regression with missing data, providing exact error characterizations.
problem High-dimensional logistic regression with missing or corrupted covariates.
method Exact characterizations of prediction and estimation errors under independence and moment conditions.
result Characterizations are universal and hold for various imputation strategies.
Strategic feature selection in high-stakes domains like healthcare.
problem Strategic manipulation of input features in algorithmic predictors.
method Formal study of strategic classification through feature selection and ridge regularization.
result Excluding individual features based on manipulability is generally suboptimal.
Optimal ridge regularization computed iteratively from generative parameters.
problem Finding the optimal ridge regularization strength for linear regression.
method Iterative procedure to compute optimal regularization strength numerically.
result The proposed procedure attains near-optimal generalization across various conditions.
Interpolating models can have heavy-tailed risk, leading to rare but severe errors.
problem Interpolating models' tail risk is poorly understood, affecting rare but impactful errors.
method Large-deviation methods to study the fragility of high-dimensional linear interpolators.
result Ridgeless regression exhibits heavy-tailed risk, while ridge-regularized estimators have better tail behavior.
A new screening method for high-dimensional data reduces computational cost.
problem Challenges in variable selection for ultrahigh-dimensional linear regression.
method Ordering absolute sample ridge partial correlations to screen variables.
result The method provides sure screening property without strong assumptions.
This article provides, through theoretical analysis, an in-depth understanding of the classification performance of the empirical risk minimization framework, in both ridge-regularized and unregularized cases, when high dimensional data are considered. Focusing on the fundamental problem of separating a two-class Gauss…
Paper proposes a novel SVM method for creating survival trees.
problem Creating non-linear survival trees for right-censored data.
method L2-regularized dipole splitting criteria with kernel methods.
result Non-linear splits using polynomial and Gaussian kernels show similar predictive power but often smaller tree sizes.
Boosting algorithms predict financial vulnerability of farmers in Chile and Tunisia.
problem Predict financial vulnerability of farmers in Chile and Tunisia using environmental data.
method Interpretable boosting algorithms based on ridge-regularized generalized linear models.
result Interaction effects improve predictive power only when included in two-step boosting.
Ridge regularized linear models (RRLMs), such as ridge regression and the SVM, are a popular group of methods that are used in conjunction with coefficient hypothesis testing to discover explanatory variables with a significant multivariate association to a response. However, many investigators are reluctant to draw ca…
Monotonic relationship found between in-distribution and out-of-distribution performance.
problem Understanding performance of machine learning models under distribution shifts.
method Analyzing ridge-regularized models and linear inverse problems under covariate shift.
result Monotonic relationship between in-distribution and out-of-distribution performance for certain models.
The paper sets fundamental limits for ERM in high dimensions.
problem Understanding statistical accuracy of ERM in high-dimensional settings.
method Sharp performance characterizations and tight lower bounds derived for generalized linear models.
result Optimal tuning of loss function and regularization parameter.
MGD with early stopping tends to ridge regularization in least squares regression.
problem Characterizing the implicit regularization of MGD with early stopping.
method Continuous-time view of MGD (momentum gradient flow) and comparison with explicit ridge regularization.
result Under optimal tuning, the risk of MGF is no more than 1.54 times that of ridge.
A conventional wisdom in statistical learning is that large models require strong regularization to prevent overfitting. Here we show that this rule can be violated by linear regression in the underdetermined n≪p situation under realistic conditions. Using simulations and real-life high-dimensional data sets, we d…
Inflating the minimum norm interpolator improves linear regression generalization error.
problem Highly anisotropic covariances and diverging d/n in linear regression. method Inflating the minimum ℓ2 norm interpolator by a constant greater than one. result Inflating the minimum norm interpolator improves generalization error.
Study addresses covariate mismatch in federated learning, improving model accuracy.
problem Learning from clients with different feature sets in federated learning.
method Developed two approaches for linear prediction under covariate mismatch: plug-in estimator and impute-then-regress strategy.
result Proposed methods provide asymptotic and finite-sample learning rates, improving model accuracy.
The study identifies spurious correlations in high-dimensional regression and quantifies their impact.
problem Spurious correlations in high-dimensional regression models.
method Statistical characterization of spurious correlations, quantifying their amount via ridge regularization.
result The value of regularization strength that minimizes test loss is in an interval where spurious correlations increase.
Optimal self-distillation improves generative models' velocity risk and mode recovery.
problem Improving generative models' velocity risk and mode recovery.
method Proved optimal self-distillation for rectified flow via linear probing, derived mixing coefficient, and provided validation tuning.
result Optimal self-distillation improves velocity risk and mode recovery.
Data coarse graining improves model performance by filtering out less relevant features.
problem Lossy data transformations lose information but can improve model generalization.
method Data coarse graining schemes that systematically discard features based on relevance to the learning task.
result A 'high-pass' scheme helps models generalize better by filtering out less relevant features.
Study shows neural collapse is invariant to class imbalances under certain conditions.
problem Neural collapse properties are only valid for balanced data.
method Adopted UFM and introduced SELI for invariant characterization.
result Embeddings and classifiers always interpolate a simplex-encoded label matrix regardless of class imbalances.
Self-training improves generalization by fitting to reliable pseudo-labels or gradually improving the classification plane.
problem Understanding how self-training improves generalization in high-dimensional Gaussian mixtures.
method Analyzing iterative self-training on binary Gaussian mixtures in the asymptotic limit.
result ST improves generalization by fitting to reliable pseudo-labels or gradually improving the classification plane.
Interpretable machine-learning models can be unstable under multicollinearity, leading to oscillatory weights that do not reflect meaningful contributions.
problem Interpretable machine-learning models can be unstable under multicollinearity.
method Theoretical analysis of eigenmodes of the feature correlation matrix.
result Small-eigenvalue modes associated with multicollinearity amplify fluctuations in the weights and generate oscillatory patterns that do not necessarily reflect meaningful contributions.
Soft diamond regularizers improve deep learning performance and sparsity.
problem Improving deep learning performance and sparsity of trained weights.
method New soft diamond synaptic weight priors based on thick-tailed symmetric alpha stable probability curves.
result Soft diamond regularizers outperform state-of-the-art methods in deep learning tasks.
Deep neural networks decompose SDF into linear and nonlinear components.
problem Constructing accurate stochastic discount factors (SDFs) for pricing.
method Additive decomposition of a deep neural network trained to construct SDFs.
result The PTK representation delivers significant performance gains in equity data.
Sparse alpha-norm regularization has many data-rich applications in Marketing and Economics. Alpha-norm, in contrast to lasso and ridge regularization, jumps to a sparse solution. This feature is attractive for ultra high-dimensional problems that occur in demand estimation and forecasting. The alpha-norm objective is …
Paper analyzes gradient descent with noisy data copies for linear regression, showing regularization and acceleration effects.
problem Improving generalization in machine learning through data augmentation with noise.
method Gradient descent with on-line noisy copies for linear regression analysis.
result Training with on-line noisy copies is equivalent to ridge regularization with a specific regularization parameter.
Modern data analysis depends increasingly on estimating models via flexible high-dimensional or nonparametric machine learning methods, where the identification of structural parameters is often challenging and untestable. In linear settings, this identification hinges on the completeness condition, which requires the …
We report on time-varying network connectedness within three banking systems: North America, the EU, and ASEAN. The original method by Diebold and Yilmaz is improved by using exponentially weighted daily returns and ridge regularization on vector autoregression (VAR) and forecast error variance decomposition (FEVD). We…
Enhances tensor regression for interpretability and performance.
problem Interpreting and modeling multidimensional tensor data with structural heterogeneity.
method Generalized Nonnegative Structured Kruskal Tensor Regression (NS-KTR) with hybrid regularization and nonnegativity constraints.
result NS-KTR outperforms conventional methods in synthetic and real hyperspectral datasets.
Framework for domain adaptation using pseudo-labels from unlabeled data.
problem Improving prediction accuracy in target domain with covariate shift.
method Kernel GLMs with labeled and pseudo-labeled data, using imputation model for target data.
result Non-asymptotic excess-risk bounds for effective labeled sample size.
PCHAL and PCHAR use principal components to speed up HAL and HAR methods.
problem Computational infeasibility in high dimensions for HAL and HAR.
method Outcome-blind principal component reduction of HAL basis.
result Empirical performance comparable to HAL and HAR, with computational gains.
Robust multivariate linear regression methods for online and offline use.
problem Estimating parameters of multivariate Gaussian linear regression models robustly.
method Robust versions of least-square criterion with online and offline algorithms.
result Asymptotic normality of robust estimates proved under weak assumptions.
Study shows neural network parameters converge to ridgelet spectrum.
problem Characterization of local minima in over-parametrized neural networks.
method Developed ridgelet transform to analyze neural network parameters.
result Distribution of parameters converges to ridgelet spectrum.
New analysis reveals optimal regularization for ESNs, avoiding double descent.
problem Characterizing and optimizing Echo State Networks (ESNs) for precise bias-variance.
method Random matrix theory applied to ESNs in a teacher-student setting.
result ESNs achieve lower MSE with limited training samples and teacher memory.
Regularization can improve both privacy and performance in machine learning models.
problem Privacy vs. Utility trade-off in machine learning models.
method The study uses logistic regression with ridge regularization and a leave-one-out analysis tool.
result Increasing the number of parameters can improve both privacy and performance when coupled with proper regularization.
Unified framework for data-driven priors in Bayesian inverse problems
problem Bayesian inverse problems
method Unified framework using score functions
result Evaluation of four data-driven priors
New insights into how randomization affects greedy model selection.
problem Understanding the impact of feature subsampling on greedy model selection.
method Investigated greedy forward selection with feature subsampling, proving effects on bias and variance.
result Ensembling with feature subsampling reduces both bias and variance, unlike convex base learners.
Inference is typically intractable in high-treewidth undirected graphical models, making maximum likelihood learning a challenge. One way to overcome this is to restrict parameters to a tractable set, most typically the set of tree-structured parameters. This paper explores an alternative notion of a tractable set, nam…
Overparameterized models are more vulnerable to membership inference attacks.
problem Vulnerability of overparameterized models to membership inference attacks.
method Theoretical and empirical analysis of overparameterized linear and ridge-regularized linear regression models in the Gaussian data setting.
result Increased number of parameters and model complexity increase vulnerability to membership inference attacks.
The study analyzes multi-class teacher-student perceptron performance and generalization errors.
problem Analyzing multi-class classification with the teacher-student perceptron.
method Deriving asymptotic expressions for Bayes-optimal and empirical risk minimization (ERM) generalization errors.
result Regularised cross-entropy minimization yields close-to-optimal accuracy for multi-class classification.
Two approaches to directly estimating Riesz representer are shown to be numerically equivalent under certain conditions.
problem Estimating Riesz representer in semiparametric statistics.
method Two distinct optimization problems solved by automatic debiased machine learning and sieve methods for conditional moment models.
result Numerical equivalence of estimators under specific regularization schemes, but not for others.