Efficiently private regression for unbounded data.
problem Privacy constraints in regression settings with unbounded covariates.
method Differential privacy techniques on mean and covariance estimation extended to sub-gaussian regime.
result Unbiased estimate of true regression vector learned up to a scaling factor.
Robustly estimates linear regression coefficients with adversarial and noisy data.
problem Estimating robust linear regression coefficients with adversarial and noisy data.
method Adversarial robust weighted Huber regression with polynomial computational complexity.
result Derives an estimation error bound that depends on the stable rank and condition number of the covariance matrix.
New method estimates covariance in deep heteroscedastic regression without labels.
problem Estimating covariance in deep heteroscedastic models is challenging due to sample-dependent covariance and lack of ground truth.
method Proposes a self-supervised approach using KL Divergence and 2-Wasserstein distance for covariance estimation and a neighborhood-based heuristic for pseudo labels.
result Demonstrates effective pseudo labels and a computationally cheaper yet accurate deep heteroscedastic regression.
Study predictive performance of linear regression with random functional covariates.
problem Theoretical predictive performance of linear regression with random functional covariates.
method Theoretical analysis of ridge and ridge-less least-squares regression with random functional covariates.
result Probabilistic bounds on predictive excess risk for random functional covariates.
Method estimates sparse inverse covariance and partial correlation matrices efficiently.
problem Sparse high-dimensional inverse covariance and partial correlation matrix estimation.
method Two-stage estimation method using partial regression with positive semi-definiteness.
result Efficient estimation of inverse covariance and partial correlation matrices with derived non-asymptotic rates.
Study on linear regression with dependent covariates, proving universality and error characterization.
problem Linear regression with dependent covariates in high-dimensional settings.
method Analysis of ridge regression performance, Gaussian universality theorem, spectral properties of covariance matrices.
result Asymptotic performance of ridge regression is invariant under non-Gaussian covariates with preserved mean and covariance.
The paper argues for using Neyman orthogonal score for balancing in debiased machine learning.
problem Debiased machine learning requires a proper approach to balance covariates.
method The paper advocates for using Riesz regression with basis functions of X for balancing.
result Covariate balancing is only valid when the score-relevant regression error is a function of covariates alone.
Novel Fréchet regression method handles errors-in-variables with low-rank covariates.
problem Regression with noisy and limited covariate data.
method Combines global Fréchet regression and principal component regression for low-rank structure.
result Improved efficiency and accuracy in high-dimensional and noisy data settings.
Improved robust regression for heavy-tailed and contaminated data.
problem Linear regression with heavy-tailed and adversarially contaminated covariates and responses.
method Applying a filtering algorithm to covariates and then using Huber regression, least trimmed squares, or least absolute deviation estimators on the remaining data.
result Near-optimal error rates achieved for the Huber regression estimator.
New similarity measure for covariate shift improves nonparametric regression rates.
problem Improving nonparametric regression under covariate shift.
method Introducing a new similarity measure based on probability ratios.
result Shows a sharper rate of convergence compared to transfer exponent.
Unified theory for causal inference using various methods.
problem Estimating causal effects in ATE estimation.
method Riesz regression, covariate balancing, DRE, TMLE, matching estimator.
result Unified theory integrating multiple methods for ATE estimation.
Study improves error bounds for sparse regression with heavy-tailed covariates.
problem Estimating sparse coefficients in linear regression with heavy-tailed covariates.
method Employed an ℓ1-penalized Huber regression method. result Error bound identical to Gaussian case for L-subexponential covariates. Study semi-supervised learning with noisy proxy covariates, deriving bounds and showing gains.
problem Learning from noisy proxy covariates with scarce labels.
method Two-stage estimator learning kernel eigenfeatures from all proxy covariates and fitting a ridge predictor on labeled data.
result Finite sample bounds show fast labeled sample rates and consistent gains over supervised and semi-supervised baselines.
This paper improves computational efficiency in kernel ridge regression under covariate shift.
problem Covariate shift in nonparametric regression.
method Random projections in RKHS to reduce computational demands.
result Significant computational savings can be achieved without compromising learning performance under covariate shift.
Improved algorithm for conditional linear regression with heterogeneous covariances.
problem Identifying a linear predictor for a fraction of data with varying covariances.
method Polynomial time algorithm using Disjunctive Normal Form (DNF) to identify a condition and linear predictor.
result Removed requirement for similar covariances in each condition term, improving algorithm applicability.
CovRegRF estimates covariance matrix from covariates using random forests.
problem Estimating conditional covariances or correlations among multivariate responses.
method Random forest trees with a custom splitting rule to maximize covariance difference.
result Accurate covariance matrix estimates and controlled Type-1 error.
GATs improve node regression on noisy graphs with provable advantage.
problem Improving node regression on graphs with noisy covariates and edges.
method Proposes a GAT designed for denoising proxy features in node regression.
result GAT achieves lower error in estimating regression coefficient and predicting responses.
Efficiently estimates sparse linear regression with heavy-tailed data and outliers.
problem Sparse estimation of linear regression coefficients with heavy-tailed covariates and noises, including outliers.
method Efficient computation of robust estimator with nearly optimal error bound.
result Nearly optimal error bound for robust sparse estimation.
Study shows pretraining and finetuning can effectively tackle covariate shift in linear regression.
problem Linear regression under covariate shift where source and target distributions differ but conditional distribution remains similar.
method Pretraining on source data and finetuning on target data using online SGD.
result Transfer learning with O(N2) source data is as effective as supervised learning with N target data. Efficiently estimates sparse linear regression with heavy-tailed and outlier-contaminated data.
problem Estimating sparse linear regression coefficients with heavy-tailed and outlier-contaminated data.
method Efficient computation of estimators with sharp error bounds.
result Sharp error bounds for efficient estimators.
Nash integrates covariate-specific side info into sparse regression via neural networks.
problem Sparse linear regression struggles with covariates exhibiting structure or coming from heterogeneous sources.
method Neural Adaptive Shrinkage (Nash) framework that integrates side information into sparse regression via neural networks. Uses split variational empirical Bayes algorithm.
result Nash improves accuracy and adaptability over existing methods in real data experiments.
A scalable algorithm for GP regression selects relevant covariates efficiently.
problem Scalable variable selection in large GP regression models.
method VGPR algorithm using Vecchia approximation for sparse precision matrix, mini-batch subsampling.
result Improved scalability and accuracy in selecting relevant covariates.
Active-set algorithm improves Cox regression for shape-restricted covariates.
problem Improving Cox regression for shape-restricted covariates.
method Shape-restricted inference using active-set optimization for spline basis expansion.
result Active-set algorithm produces accurate linear covariate effect estimates.
Network-assisted regression uses conformal prediction for valid inference.
problem Predicting node attributes using network and conventional covariates with valid statistical inference.
method Network analog of conformal prediction under mild joint exchangeability assumption.
result Achieves finite sample validity and asymptotic conditional validity for various network covariates.
Sparse models for high-dimensional linear regression and machine learning have received substantial attention over the past two decades. Model selection, or determining which features or covariates are the best explanatory variables, is critical to the interpretability of a learned model. Much of the current literature…
Proposes a robust method for predicting missing outcomes in covariate shift adaptation.
problem Predicting missing outcomes in test data with covariate shift.
method Doubly robust estimator for covariate shift adaptation via importance weighting, incorporating an additional estimator for the regression function.
result Shows robustness against density-ratio estimation errors, maintaining consistency if either estimator is consistent.
Optimally tackles covariate shift in RKHS-based nonparametric regression.
problem Covariate shift in nonparametric regression over RKHS.
method Two families of covariate shift problems defined using likelihood ratios. Minimax rate-optimal estimators for KRR and reweighted KRR.
result KRR is minimax rate-optimal and strictly sub-optimal compared to naive estimator under covariate shift.
Paper develops statistical tests for covariance matrix regression on manifold.
problem Regression with random covariance matrices in Fréchet space.
method Develops Wasserstein F-tests for Bures-Wasserstein manifold.
result Asymptotic null distribution and power of the test.
Develops a method for kernel ridge regression under covariate shift using pseudo-labels.
problem Learning a regression function with small mean squared error over a target distribution with labeled data from a different feature distribution.
method Split labeled data into two subsets, conduct kernel ridge regression on each, use imputation model to fill missing labels, and select the best candidate model.
result Non-asymptotic excess risk bounds demonstrate effective adaptation to target distribution and covariate shift.
New algorithm mitigates misspecification amplification in regression models with covariate shift.
problem Distribution shift and model misspecification in regression models.
method Developed a new algorithm inspired by robust optimization to avoid misspecification amplification.
result No misspecification amplification while still achieving optimal statistical rates.
Proposes FarmHazard model for hazard regression with correlated covariates.
problem Model selection challenges in high-dimensional data with correlated covariates.
method Factor-Augmented Regularized Model for Hazard Regression (FarmHazard) that learns latent factors and idiosyncratic components.
result Proves model selection and estimation consistency under mild conditions.
Improves regression models' performance on covariate shift.
problem Out-of-distribution generalization for regression.
method Spectrally adapting the weights of a pre-trained neural regression model.
result Spectral adaptation improves out-of-distribution performance.
Paper addresses covariate shift in deep learning regression models.
problem Covariate shift in dependent data from different distributions.
method Sparse-penalized deep neural network (SPDNN) estimator for nonparametric regression.
result Adaptive convergence rates for quantile and Huber regression.
Multivariate regression model is a natural generalization of the classical univari- ate regression model for fitting multiple responses. In this paper, we propose a high- dimensional multivariate conditional regression model for constructing sparse estimates of the multivariate regression coefficient matrix that accoun…
Bayesian methods estimate regression functions on submanifolds using graph Laplacian eigenbasis.
problem Estimating regression functions on unknown smooth submanifolds.
method Random geometric graph structure, Bayesian priors based on random basis expansion in graph Laplacian eigenbasis.
result Posterior contraction rates are minimax optimal for any positive smoothness index.
Deep model predicts shapes of curves with multiple covariates.
problem Predicting shapes of planar curves with various covariates.
method Deep learning model using complex-valued functions, conditional covariance smoother with modality-specific encoders.
result Model accurately predicts shapes of curves with multimodal covariates.
New method for private linear regression under privacy constraints, achieving optimal rates.
problem Statistical complexity of private linear regression under unknown, ill-conditioned covariates.
method Information-Weighted Regression method
result Optimal convergence rates for both central and local privacy models.
New hierarchical model improves on standard practice for high-dimensional data.
problem Poor statistical performance in high-dimensional hierarchical models.
method Model effects as exchangeable across covariates and correlated across datasets.
result Empirical Bayes estimator outperforms classic approach in high-dimensional settings.
Sliced inverse regression is a popular tool for sufficient dimension reduction, which replaces covariates with a minimal set of their linear combinations without loss of information on the conditional distribution of the response given the covariates. The estimated linear combinations include all covariates, making res…
Unified approach to linear regression using covariance fitting for optimal weights.
problem Finding optimal weights for linear regression models when weights are unknown.
method Covariance fitting SPICE-methodology to obtain data-adaptive weights.
result Tuned versions of known regularized estimators are unified under a common approach.
Efficiently estimates prediction error in regression with Gaussian covariates under privacy constraints.
problem Private regression with Gaussian covariates under differential privacy constraints.
method Sum-of-Squares framework combined with robust estimators.
result Sample-optimal private regression algorithm with optimal error rates.
New method for causal inference with observed covariates improves learning rates.
problem Causal inference with observed covariates in nonparametric instrumental variable regression.
method Introduces novel Fourier measure for partial smoothing and adapts kernel lengthscales for anisotropic smoothness.
result Upper and lower learning rates for KIV-O show interpolation between NPIV and NPR rates.
Maximum likelihood estimator performance in logistic regression analyzed.
problem Performance of maximum likelihood estimator in logistic regression.
method Sharp non-asymptotic guarantees for existence and excess logistic risk.
result Sharp guarantees for the existence and excess risk of MLE in logistic regression.
Forward regression is a statistical model selection and estimation procedure which inductively selects covariates that add predictive power into a working statistical regression model. Once a model is selected, unknown regression parameters are estimated by least squares. This paper analyzes forward regression in high-…
Proposes a method to improve regression model performance with limited target data using fused-regularizer.
problem Model shifts and covariate shifts in high-dimensional regression.
method Two-step method with fused-regularizer to leverage source data for target task.
result Robust to covariate shifts, minimax-optimal under certain conditions, and validated by numerical tests.
Machine learning improves joint default assessment by capturing non-linear dependencies.
problem Capturing non-linear dependencies among covariates for accurate joint default assessment.
method Application of machine learning techniques to credit card dataset, comparing with logistic regression.
result Machine learning outperforms logistic regression in assessing portfolio riskiness.
The application of standard sufficient dimension reduction methods for reducing the dimension space of predictors without losing regression information requires inverting the covariance matrix of the predictors. This has posed a number of challenges especially when analyzing high-dimensional data sets in which the numb…
Covariance-Driven Regression Trees reduce overfitting in CART.
problem Overfitting in CART decision trees, especially with small sample sizes.
method Covariance-driven splitting criterion for regression trees (CovRT).
result CovRT achieves superior prediction accuracy compared to CART in simulations and real-world tasks.