Paper tackles high-dimensional quantile regression with distribution shift using transfer learning.
problem Efficiency of knowledge transfer is severely impacted by distribution shift in high-dimensional regression.
method Proposes a novel transferable set and framework for three types of distribution shift: parameter, covariate, and residual.
result Establishes estimation error bounds and source detection consistency for the proposed method.
Develops estimators for near-optimal linear regression under distribution shift.
problem Linear regression under distribution shift with scarce target domain data.
method Minimax linear risk estimators covering various transfer learning settings.
result Achieves near-optimal risk for linear regression problems under distribution shift.
Task shift from classification to regression is possible in overparameterized linear models with limited additional data.
problem Transferability of latent knowledge from classification to regression in overparameterized linear models.
method Investigation of task shift in overparameterized linear regression, zero-shot and few-shot cases, with a focus on minimum-norm interpolation.
result Minimum-norm interpolators can transfer latent knowledge from classification to regression with limited additional data.
New algorithm mitigates misspecification amplification in regression models with covariate shift.
problem Distribution shift and model misspecification in regression models.
method Developed a new algorithm inspired by robust optimization to avoid misspecification amplification.
result No misspecification amplification while still achieving optimal statistical rates.
New similarity measure for covariate shift improves nonparametric regression rates.
problem Improving nonparametric regression under covariate shift.
method Introducing a new similarity measure based on probability ratios.
result Shows a sharper rate of convergence compared to transfer exponent.
Analysis of ridge regression under concept shift reveals nontrivial effects on generalization performance.
problem Understanding and mitigating the impact of distribution shift in machine learning models.
method Derivation of exact prediction risk expression in the thermodynamic limit for ridge regression under concept shift.
result Reveals a phase transition and nonmonotonic data dependence of test performance under concept shift.
Develops an MS-inspired algorithm for regression mode finding and space partitioning.
problem Finding local modes of regression functions and partitioning input space.
method Mean-shift-inspired algorithm for iterative gradient ascent.
result Proves convergence and rates of convergence for estimated local modes.
This paper improves computational efficiency in kernel ridge regression under covariate shift.
problem Covariate shift in nonparametric regression.
method Random projections in RKHS to reduce computational demands.
result Significant computational savings can be achieved without compromising learning performance under covariate shift.
Corrects distribution shift in target shift scenarios using importance weighting.
problem Analyzes importance weighting for correcting distribution shift under target shift.
method Analyzed importance-weighted kernel ridge regression under target shift.
result Shows that importance weighting corrects the train-test mismatch without altering input-space complexity.
Proposes a method to improve regression model performance with limited target data using fused-regularizer.
problem Model shifts and covariate shifts in high-dimensional regression.
method Two-step method with fused-regularizer to leverage source data for target task.
result Robust to covariate shifts, minimax-optimal under certain conditions, and validated by numerical tests.
Optimally tackles covariate shift in RKHS-based nonparametric regression.
problem Covariate shift in nonparametric regression over RKHS.
method Two families of covariate shift problems defined using likelihood ratios. Minimax rate-optimal estimators for KRR and reweighted KRR.
result KRR is minimax rate-optimal and strictly sub-optimal compared to naive estimator under covariate shift.
Density-Regression improves deep uncertainty estimation with faster inference.
problem Efficient uncertainty estimation under distribution shifts with modern deep models.
method Leverages density function for fast inference and distance-aware feature space.
result Density-Regression achieves competitive uncertainty estimation performance.
Develops a method for kernel ridge regression under covariate shift using pseudo-labels.
problem Learning a regression function with small mean squared error over a target distribution with labeled data from a different feature distribution.
method Split labeled data into two subsets, conduct kernel ridge regression on each, use imputation model to fill missing labels, and select the best candidate model.
result Non-asymptotic excess risk bounds demonstrate effective adaptation to target distribution and covariate shift.
New insights into how high-dimensional models handle covariate shifts.
problem Covariate shift in high-dimensional random feature regression.
method Exact high-dimensional asymptotics of random feature regression under covariate shift.
result Overparameterized models exhibit enhanced robustness to covariate shift.
This research improves online learning by correcting for target shift in machine learning.
problem Online learning struggles with distributional shift, especially in target values.
method Derives closed-form expressions for online and offline learning, and target correction.
result Online kernel-based learning can learn the same predictor as offline learning with target correction.
Extends FJS analysis to general label spaces, including classification and regression.
problem Distribution shift in general label spaces, including covariate and label shifts.
method Proposes a framework for analyzing FJS in general label spaces and generalizes existing results.
result Generalizes FJS analysis to general label spaces, including classification and regression.
Improves regression models' performance on covariate shift.
problem Out-of-distribution generalization for regression.
method Spectrally adapting the weights of a pre-trained neural regression model.
result Spectral adaptation improves out-of-distribution performance.
ReTaSA tackles continuous target shift in regression problems.
problem Continuous target shift in regression settings.
method Nonparametric regularized approach to estimate importance weight function.
result The method provides theoretical justification for the estimated importance weight function.
Study shows pretraining and finetuning can effectively tackle covariate shift in linear regression.
problem Linear regression under covariate shift where source and target distributions differ but conditional distribution remains similar.
method Pretraining on source data and finetuning on target data using online SGD.
result Transfer learning with O(N2) source data is as effective as supervised learning with N target data. Paper addresses covariate shift in deep learning regression models.
problem Covariate shift in dependent data from different distributions.
method Sparse-penalized deep neural network (SPDNN) estimator for nonparametric regression.
result Adaptive convergence rates for quantile and Huber regression.
The paper analyzes covariate shift in nonparametric regression with Markovian data.
problem Covariate shift in regression problems with Markovian data.
method Extension of nonparametric convergence rates to Markovian dependence structures, using Hölder smoothness assumptions and similarity measures.
result Precise convergence rates for Nadaraya-Watson kernel estimators under specific Markovian conditions.
This paper examines how adversarial perturbations affect model performance and equilibrium learning.
problem Adversarial perturbations and covariate shifts impact model performance and equilibrium learning.
method Characterizes the extrapolation region in regression and classification, analyzes dynamics of adversarial learning games.
result Establishes two directional convergence results: a blessing in regression and a curse in classification.
Quick Shift is a popular mode-seeking and clustering algorithm. We present finite sample statistical consistency guarantees for Quick Shift on mode and cluster recovery under mild distributional assumptions. We then apply our results to construct a consistent modal regression algorithm.
Study optimal ridge regularization for out-of-distribution prediction.
problem Optimal ridge regularization for predicting out-of-distribution data.
method Established conditions for optimal regularization under covariate and regression shifts, proving monotonic risk in data aspect ratio.
result Negative regularization can be optimal under shifts, even with isotropic or underparameterized training features.
Proposes a robust method for predicting missing outcomes in covariate shift adaptation.
problem Predicting missing outcomes in test data with covariate shift.
method Doubly robust estimator for covariate shift adaptation via importance weighting, incorporating an additional estimator for the regression function.
result Shows robustness against density-ratio estimation errors, maintaining consistency if either estimator is consistent.
Improved logistic regression for robustness to distribution shifts.
problem Distribution shifts in social and behavioral sciences.
method Distributionally robust logistic regression with graph-based solution.
result Significant improvement in calibration and AUC metrics.
FDN improves probabilistic regressors' adaptability to distribution shifts.
problem Overconfidence in modern probabilistic regressors under distribution shift.
method FDN uses input-conditioned distributions over network weights, trained with a Monte Carlo beta-ELBO objective.
result FDN produces predictive mixtures whose dispersion adapts to the input, providing shift-aware uncertainty.
Paper tackles dynamic label shift in online learning, achieving optimal performance.
problem Adapting to changing class marginals in online supervised and unsupervised learning.
method Develops novel algorithms reducing adaptation to online regression, achieving optimal dynamic regret.
result Achieves superior performance in various online label shift scenarios.
Regularizes ML algorithms for robust multivariate analysis against distribution shifts.
problem Ensuring robustness of multivariate analysis algorithms against distribution shifts.
method Integrates a causal regularisation term into the loss function of multivariate analysis algorithms.
result Demonstrates improved out-of-distribution generalisation with reduced-rank regression and partial least squares.
Study on how kernel regression models generalize to out-of-distribution data.
problem Understanding generalization in machine learning models under distributional shifts.
method Replica method from statistical physics to derive analytical formula for generalization error.
result Identified overlap matrix as key determinant of generalization performance under distribution shift.
A robust method for off-policy evaluation in contextual bandits.
problem Evaluating policies when direct methods are unavailable.
method Robust regression approach to off-policy evaluation.
result Superior empirical performance across benchmarks.
Study investigates OOD generalization methods for mechanics problems.
problem Real-world mechanics problems with unknown test environments and data distribution shifts.
method Investigates OOD generalization methods for regression problems in mechanics.
result OOD generalization methods perform better than traditional ML methods on mechanics-specific regression problems.
New federated conformal prediction method addresses label shift for uncertainty quantification.
problem Label shift in federated learning and its impact on uncertainty quantification.
method Quantile regression-based federated conformal prediction method with privacy constraints.
result Method provides valid coverage of prediction sets and differential privacy guarantees.
Paper analyzes spectral algorithms under covariate shift, providing convergence rates.
problem Addressing distributional mismatch in regression models.
method Incorporates importance weights into spectral algorithms in RKHS.
result Establishes minimax-optimal convergence rates for misspecified cases.
New method tackles concept shifts in nonparametric regression using robust and adaptive transfer learning.
problem Concept shifts and sample scarcity in target domains hinder nonparametric regression.
method Robust and adaptive transfer learning procedure leveraging fixed bandwidth Gaussian kernels.
result Spectral algorithms with fixed bandwidth Gaussian kernels attain minimax convergence rates for nonparametric regression.
Proposes a method to use external machine-learning predictions in multinomial logistic regression.
problem Improving statistical inference using summary-level external machine-learning predictions.
method Empirical-likelihood framework incorporating moment constraints from external nonparametric machine-learning predictions.
result Fused estimator achieves strict efficiency gain over primary-only estimator under mild conditions.
The paper explores optimal algorithms for linear regression under covariate shift, proving the optimality of certain transformations and SGD variants.
problem Optimal algorithms for linear regression under covariate shift with ellipse-shaped constraints.
method Establishes a tight lower generalization bound via Bayesian Cramer-Rao inequality, proves the optimality of certain transformations, and analyzes SGD variants.
result Optimal estimators and SGD variants achieve optimality under specific conditions.
The paper analyzes how re-weighting helps in reducing variance in high-dimensional kernel methods under covariate shifts.
problem The challenge of high-dimensional kernel methods under covariate shifts and the role of re-weighting.
method Derives asymptotic expansion of high-dimensional kernels under covariate shifts, analyzes bias-variance decomposition, and characterizes the regularized kernel.
result Re-weighting helps in decreasing variance and can be seen as a data-dependent regularization.
Paper develops estimators for unbounded density ratios with applications in error control.
problem Estimating density ratios with unbounded domains and ranges.
method Least squares and logistic regression loss functions for density ratio estimation.
result Established upper bounds on estimation errors with optimal rates for unbounded density ratios.
New CPS model tackles conditional probability shift in machine learning.
problem Discrepancy between source and target distributions in machine learning.
method Conditional Probability Shift Model (CPSM) using multinomial regression and EM algorithm.
result Superior balanced classification accuracy on target data compared to existing methods.
Improves prediction stability with model misspecification and distribution shift.
problem Inaccurate parameter estimation and instability of prediction in real-world applications.
method Proposes Decorrelated Weighting Regression (DWR) algorithm to optimize weights for samples and variables.
result Significantly improves accuracy of parameter estimation and prediction stability.
Method constructs nonparametric prediction intervals with finite-sample guarantees.
problem Nonparametric instrumental variable regression with finite-sample coverage.
method Conformal inference framework applied to NPIV, combining with various estimators.
result Distribution-free, finite-sample coverage over chosen IV shifts.
Nonlinear kernel regression models are often used in statistics and machine learning because they are more accurate than linear models. Variable selection for kernel regression models is a challenge partly because, unlike the linear regression setting, there is no clear concept of an effect size for regression coeffici…
Transformers show better in-context learning resilience under distribution shifts than simple MLPs.
problem Understanding in-context learning under varying distribution shifts.
method Comparing transformers and set-based MLPs on linear regression tasks.
result Transformers better emulate OLS performance and exhibit better resilience to mild distribution shifts.
Sharp analysis of knowledge distillation for high-dimensional regression.
problem Characterizing the risk of target models in high-dimensional settings.
method Sharp non-asymptotic bounds for ridgeless regression under model and distribution shifts.
result Identifies optimal surrogate models and reveals benefits and limitations of discarding weak features.
Algorithm adapts to shifting domains with minimal label queries.
problem Adaptive learning in online machine learning systems with domain shifts.
method Adaptive algorithm balancing regret and label queries for hidden domains.
result Achieves lower regret compared to uniform and greedy queries.
Paper tackles unbounded density ratio estimation for covariate shift adaptation.
problem Understudied challenge in statistical learning: unbounded density ratios.
method Three-step estimation method: relative density ratio, truncation, and transformation.
result Established rigorous convergence guarantees for density ratio and regression estimators.
Unified analysis of kernel-based methods under covariate shift.
problem Covariate shift in learning problems.
method Unified analysis of nonparametric methods in RKHS.
result Sharp convergence rates for general loss functions.