Regularized linear regression improves binary classification performance, especially with ridge and ℓ1 regularization.
problem Improving binary classification accuracy with noisy labels.
method Systematic study of regularization strengths on linear classifiers trained on noisy binary classification data.
result Ridge regression consistently improves classification error, while ℓ1 regularization can induce sparsity and ℓ∞ regularization can concentrate weights to two values. The nullspace and regularization impact high-dimensional linear regression interpretability.
problem Interpreting high-dimensional linear regression coefficients in complex data.
method Optimization formulation to compare coefficients and physical knowledge.
result Regularization and z-scoring choices affect interpretability and true coefficient closeness.
Unified approach to linear regression using covariance fitting for optimal weights.
problem Finding optimal weights for linear regression models when weights are unknown.
method Covariance fitting SPICE-methodology to obtain data-adaptive weights.
result Tuned versions of known regularized estimators are unified under a common approach.
Efficiently performs robust and sparse kernel regression.
problem Robust and sparse kernel regression.
method Sign gradient descent and early stopping.
result Sign gradient descent achieves robust and sparse kernel regression efficiently.
Regularization aims to improve prediction performance of a given statistical modeling approach by moving to a second approach which achieves worse training error but is expected to have fewer degrees of freedom, i.e., better agreement between training and prediction error. We show here, however, that this expected beha…
A new method for linear regression using feature graphs and hierarchical shrinkage.
problem Estimating robust parameters for linear regression models.
method Hierarchical Feature Regression (HFR) estimator that constructs a supervised feature graph to shrink parameters towards group targets.
result Demonstrates good predictive accuracy and versatility compared to other regularization techniques.
We simplify complex regression coefficients using linearization and feature comparison.
problem Interpreting high-dimensional regression coefficients from nonlinear responses.
method Developed a linearization method to derive feature coefficients and compare them with regression coefficients.
result Shows how regression coefficients relate to linearized feature coefficients and how they change under regularization.
We present a detailed analysis of the class of regression decision tree algorithms which employ a regulized piecewise-linear node-splitting criterion and have regularized linear models at the leaves. From a theoretic standpoint, based on Rademacher complexity framework, we present new high-probability upper bounds for …
GD outperforms ridge regression and SGD in linear regression problems.
problem Comparing the risks of GD, ridge regression, and SGD in linear regression problems.
method Instance-wise finite-sample risk analysis of GD, ridge regression, and SGD.
result GD outperforms ridge regression and is incomparable with SGD in some cases.
In this paper, we investigate a multivariate multi-response (MVMR) linear regression problem, which contains multiple linear regression models with differently distributed design matrices, and different regression and output vectors. The goal is to recover the support union of all regression vectors using l1/l2-reg…
SGD implicitly regularizes linear regression problems better than ridge regression for many cases.
problem Understanding implicit regularization in linear regression problems.
method Comparing SGD and ridge regression on a broad class of least squares problems.
result SGD generalizes no worse than ridge regression for many problem instances, sometimes better.
The notion of developing statistical methods in machine learning which are robust to adversarial perturbations in the underlying data has been the subject of increasing interest in recent years. A common feature of this work is that the adversarial robustification often corresponds exactly to regularization methods whi…
New scalable algorithm for non-negative linear regression with entropy-regularized OT loss.
problem Generalizing task-specific linear models to broader applications.
method Sinkhorn-like scaling iterations for convex penalty and datafit terms.
result Simple multiplicative updates for various penalty and datafit terms.
Enhances GBDT robustness with one-hot encoding and regularization.
problem Low robustness of GBDT models against covariate perturbation.
method One-hot encoding to linear framework, risk decomposition, L1 or L2 regularization. result Regularization enhances GBDT robustness.
Enhances KLR for indefinite kernels with L1-norm regularization.
problem Classifying with indefinite kernels captures more domain-specific information.
method Introduces L1-norm regularization to induce sparsity and a proximal linearized algorithm. result Superior performance in accuracy and sparsity on multiple datasets.
The paper proposes a gradient-based method for multi-penalty Ridge regression.
problem Optimizing multiple regularization hyperparameters for linear regression.
method Gradient-based optimization through matrix differential calculus.
result The method outperforms traditional regularization techniques like LASSO and Ridge.
New model leads to optimal test loss in sparse linear regression.
problem Sparse linear regression with low test loss despite interpolating training data.
method Developed a new parametrization of the model that combines benefits of ℓ1 and ℓ2 norms.
result Training via gradient descent leads to an interpolator with near-optimal test loss.
Many statistical estimators for high-dimensional linear regression are M-estimators, formed through minimizing a data-dependent square loss function plus a regularizer. This work considers a new class of estimators implicitly defined through a discretized gradient dynamic system under overparameterization. We show that…
This research uses DPPs to improve semi-parametric regression models.
problem Improving comprehensibility in semi-parametric regression models without sacrificing accuracy.
method Introduced a novel representation of finite DPPs and used it to derive a key identity illustrating implicit regularization.
result Demonstrated the implicit regularization effect of determinantal sampling for semi-parametric regression.
The paper analyzes early stopping in linear regression and shows it's equivalent to ridge regularization.
problem Understanding the effect of early stopping on linear regression models.
method Characterization of gradient descent dynamics and analysis of excess risk.
result Early stopped solution is equivalent to minimum norm solution for a generalized ridge regularized problem.
Optimal regularization can prevent the double descent phenomenon in learning models.
problem The double descent phenomenon in learning models, where test performance is non-monotonic in sample size and model size.
method Theoretical and empirical study of optimal ℓ2 regularization for linear regression models and neural networks. result Optimally-tuned ℓ2 regularization achieves monotonic test performance for certain models and mitigates the double descent phenomenon for more general models. Optimal ridge regularization computed iteratively from generative parameters.
problem Finding the optimal ridge regularization strength for linear regression.
method Iterative procedure to compute optimal regularization strength numerically.
result The proposed procedure attains near-optimal generalization across various conditions.
Gradient descent implicitly favors group sparsity in neural networks.
problem Understanding implicit regularization in neural networks for structured sparsity.
method Novel neural reparameterization for diagonally grouped linear networks.
result Gradient descent without explicit regularization biases towards group sparsity.
Study on deleting user data in linear regression models to maintain limited memory.
problem Deleting user data in a limited time frame for statistical models.
method Proposed FIFD-OLS and FIFD-Adaptive Ridge algorithms for low-dimensional and online settings.
result Demonstrated effectiveness of FIFD-Adaptive Ridge in maintaining statistical efficiency.
NA0CT2 improves tensor regression predictions with ℓ0 regularization.
problem Improving tensor regression predictions with structural information.
method Noise-Augmented ℓ0 regularization on Tucker decomposition. result Achieves exact ℓ0 regularization on core tensor in linear and generalized linear tensor regression. Private sketches protect linear regression data privacy.
problem Protecting sensitive information in linear regression.
method Release private sketches of datasets, compute approximate solutions.
result Private sketches maintain good approximation guarantees to the original problem.
We analyze optimal weighted ridge regression in overparameterized linear models.
problem Optimal regularization in overparameterized linear regression models.
method Generalized ridge regression with weighted regularization.
result The optimal regularization parameter can be negative in overparameterized settings.
Study optimizes linear regression analysis for high-dimensional settings.
problem Understanding high-dimensional linear regression with interpolation and regularization.
method Localized uniform convergence analysis of optimistic rates for linear regression.
result Recover guarantees for ridge and LASSO regression under random designs.
This work analyzes how transformers learn common linear regression tasks.
problem Understanding how in-context learning operates in real-world applications with common task structures.
method Analyzing a linear attention model trained on low-rank regression tasks.
result Statistical fluctuations in finite pre-training data induce an implicit regularization, leading to a sharp phase transition in generalization error.
Unified framework for shrinkage, thresholding, and regularization in normal mean estimation and linear regression.
problem Estimation of normal mean in multivariate settings with correlated observations.
method Approximate risk minimization over a functional class of shrinkage-thresholding rules.
result Unified estimator NOMAD for shrinkage, thresholding, and regularization.
The paper analyzes how gradient descent implicitly regularizes solutions in overparameterized neural networks, revealing depth-dependent regularization effects.
problem Understanding implicit regularization in overparameterized linear neural networks for regression problems.
method Analyzing the approximation error between gradient flow limit points and ℓ1-minimization solutions, deriving tight upper and lower bounds. result The approximation error decreases linearly for D≥3 and at a slower rate for D=2, linked to null space property constants. Improved regression analysis using Padé approximants with new residuals and regularization.
problem Improving regression analysis with Padé approximants for accuracy and avoiding overfitting.
method New residuals in least squares method, system of linear equations for rational functions, Tikhonov regularization.
result Demonstrated efficiency in practical cases from physics and reliability theory.
Study on linear regression with dependent covariates, proving universality and error characterization.
problem Linear regression with dependent covariates in high-dimensional settings.
method Analysis of ridge regression performance, Gaussian universality theorem, spectral properties of covariance matrices.
result Asymptotic performance of ridge regression is invariant under non-Gaussian covariates with preserved mean and covariance.
This paper analyzes divide-and-conquer estimators for functional linear regression without assuming target function in the RKHS.
problem Functional linear regression without target function in RKHS.
method Integral operator approach to establish upper bounds and prove asymptotic optimality.
result Sharp finite sample upper bounds and asymptotic optimality of divide-and-conquer estimators.
Paper reformulates UOT as non-negative penalized linear regression for efficient algorithms.
problem Optimal transport with relaxed marginal conditions.
method Reformulate UOT as non-negative penalized linear regression, propose multiplicative updates.
result Efficient algorithms for UOT with quadratic penalties, continuity of solutions.
In this paper, we consider stochastic dual coordinate (SDCA) {\em without} strongly convex assumption or convex assumption. We show that SDCA converges linearly under mild conditions termed restricted strong convexity. This covers a wide array of popular statistical models including Lasso, group Lasso, and logistic reg…
Paper proposes an online sparse linear regression method for streaming data.
problem Sparse regression for variable selection and prediction accuracy.
method Online sparse linear regression framework with memory efficiency and relaxed assumptions.
result The ℓ2-norm statistical error of the estimator diminishes to zero with optimal order. The paper analyzes ℓ1-LinR for Ising model selection using statistical mechanics.
problem Model selection consistency of ℓ1-LinR for Ising models. method Replica method from statistical mechanics, ℓ1-regularized linear regression (ℓ1-LinR). result Model selection consistency with sample complexity $M=\mathcal{O}\left(\log N
ight)$.
Recent research has studied the role of sparsity in high dimensional regression and signal reconstruction, establishing theoretical limits for recovering sparse models from sparse data. This line of work shows that ℓ1-regularized least squares regression can accurately estimate a sparse linear model from n nois…
Inflating the minimum norm interpolator improves linear regression generalization error.
problem Highly anisotropic covariances and diverging d/n in linear regression. method Inflating the minimum ℓ2 norm interpolator by a constant greater than one. result Inflating the minimum norm interpolator improves generalization error.
Method estimates parameters of complex nonlinear systems.
problem Parameter estimation for nonlinear systems with derivative states.
method Regularized linear regression using differentiation filtering and least squares.
result Finite-sample bound on mean absolute error of estimation.
Method introduces topological regularization using information filtering networks.
problem Sparse probabilistic modeling and multicollinear regression.
method Topological regularization via information filtering network.
result Direct application to L0-norm regularized problems. Improved sketching for logistic and ℓ1 regression with near-linear dimensions.
problem Efficiently approximate ℓ1 and logistic regression problems. method New sketching techniques achieving near-linear dimensions for both problems.
result Achieved near-linear sketching dimensions for ℓ1 and logistic regression. We study the following three fundamental problems about ridge regression: (1) what is the structure of the estimator? (2) how to correctly use cross-validation to choose the regularization parameter? and (3) how to accelerate computation without losing too much accuracy? We consider the three problems in a unified larg…
Paper tackles functional linear regression using spectral algorithms with discrete observations.
problem Functional linear regression problem with discretely observed data.
method Combines distributed spectral algorithms with Sobolev kernels for regularization.
result Derives matching upper and lower bounds for convergence in Sobolev norm.
The paper compares traditional regression with modern neural network methods for financial hedging and risk compression.
problem Finding optimal hedge ratios and managing portfolio risk using traditional regression methods has limitations.
method The paper introduces regularization techniques and common factor analyses using neural networks to improve upon regression methods.
result Neural network methods provide better performance in hedge ratio estimation and risk compression compared to traditional regression.
Picasso is a new library for sparse learning problems in R and Python.
problem Sparse learning problems in high-dimensional data analysis.
method Unified framework of pathwise coordinate optimization with efficient active set selection strategies.
result picasso can efficiently handle large-scale problems.
Estimates linear model from noisy covariates and instruments using spectral regularization.
problem Estimating a linear model from many noisy covariates and instruments.
method Two-stage least squares with spectral regularization of canonical correlations.
result Upper and lower bounds on estimation error, proving optimality of the method with noisy data.