SGD improves DR by solving two-stage sampling problems.
problem Improving the learning properties of SGD for distribution regression.
method Applying SGD to two-stage sampling problems in distribution regression.
result Theoretical guarantees for SGD's performance in DR, with optimal bounds.
The paper analyzes how generated data improves adversarial training in high-dimensional regression.
problem Improving adversarial training in high-dimensional regression.
method Theoretical analysis of a two-stage training approach with generated data and pseudo-labels.
result Two-stage adversarial training achieves better performance than ridgeless training in high-dimensional linear regression.
Improved learning theory for kernel distribution regression with two-stage sampling.
problem Distribution regression problem and two-stage sampling setting.
method Kernel methods, near-unbiased condition, new error bounds, convergence rates.
result Strictly improved convergence rates for three important classes of kernels.
A new framework integrates classification and regression tasks in multi-task learning.
problem Jointly solving classification and regression tasks in multi-task scenarios.
method Two-Stage Learning-to-Defer (L2D) framework with a unified deferral mechanism.
result Unified deferral mechanism ensures convergence to the Bayes-optimal rejector.
New algorithm protects privacy in IVaR regression while maintaining accuracy.
problem Privacy leakage in classical IVaR methods.
method Noisy two-stage gradient descent with differential privacy guarantees.
result Achieves statistical efficiency and privacy in IVaR regression.
We present a novel algorithm for non-linear instrumental variable (IV) regression, DualIV, which simplifies traditional two-stage methods via a dual formulation. Inspired by problems in stochastic programming, we show that two-stage procedures for non-linear IV regression can be reformulated as a convex-concave saddle-…
We propose a novel method designed for large-scale regression problems, namely the two-stage best-scored random forest (TBRF). "Best-scored" means to select one regression tree with the best empirical performance out of a certain number of purely random regression tree candidates, and "two-stage" means to divide the or…
POTEC tackles off-policy learning in large action spaces, improving effectiveness.
problem Existing OPL methods fail in large discrete action spaces due to bias or variance issues.
method Two-stage algorithm: cluster selection via policy-based approach, action selection via regression-based approach.
result POTEC provides substantial improvements in off-policy learning effectiveness, especially in large and structured action spaces.
SARD improves adversarial robustness in two-stage L2D systems.
problem Adversarial attacks can manipulate query allocation in two-stage L2D systems.
method Introduces SARD, a convex learning algorithm with provable guarantees.
result SARD significantly improves robustness under adversarial attacks while maintaining strong clean performance.
Fast nonparametric conditional independence testing via two-stage regression
problem Fast nonparametric conditional independence testing
method BLITZ (Broad-to-Local Independence Testing via residualiZation)
result Better null calibration than fast kernel, random-feature, and regression-based competitors
New method detects biomarker-treatment interactions in clinical trials.
problem Detecting interactions between high-dimensional biomarkers and treatments in randomized trials.
method Two-stage penalized regression screening using ridge regression for multivariate screening.
result Ridge regression screening provides greater power than traditional methods in correlated data.
Paper develops a new method for distribution regression with indefinite kernels.
problem Distribution regression with indefinite kernels.
method Coefficient-based regularized distribution regression with two-stage sampling.
result Optimal learning rates derived for the algorithm under mild conditions.
We focus on the distribution regression problem: regressing to a real-valued response from a probability distribution. Although there exist a large number of similarity measures between distributions, very little is known about their generalization performance in specific learning tasks. Learning problems formulated on…
Paper introduces robust distribution regression using kernel methods.
problem Distribution regression from probability measures to real-valued responses.
method Introduces a robust loss function lσ and a windowing function V for two-stage sampling problems. result Shows improved learning rates and robustness with the robust distribution regression (RDR) scheme.
We propose a new two stage algorithm LING for large scale regression problems. LING has the same risk as the well known Ridge Regression under the fixed design setting and can be computed much faster. Our experiments have shown that LING performs well in terms of both prediction accuracy and computational efficiency co…
DFIV uses deep features for IV regression, achieving optimal rates.
problem Optimal IV regression with deep features for complex target functions.
method Two-stage approach: deep feature learning followed by IV regression.
result DFIV achieves minimax optimal learning rate under certain conditions.
Algorithmic decision making process now affects many aspects of our lives. Standard tools for machine learning, such as classification and regression, are subject to the bias in data, and thus direct application of such off-the-shelf tools could lead to a specific group being unfairly discriminated. Removing sensitive …
A tacit assumption in linear regression is that (response, predictor)-pairs correspond to identical observational units. A series of recent works have studied scenarios in which this assumption is violated under terms such as ``Unlabeled Sensing and ``Regression with Unknown Permutation''. In this paper, we study the s…
A two-stage GPR framework with automatic kernel search and subsampling improves prediction accuracy and efficiency.
problem Inaccurate predictions due to misspecified mean and kernel functions in Gaussian Process Regression.
method Two-stage GPR, automatic kernel search, subsampling for hyperparameter initialization.
result Competitive or better performance compared to full dataset training, robust on real-world datasets.
DML-IV improves IV regression for learning decision policies by reducing bias.
problem Spurious correlations in offline datasets caused by hidden confounders.
method Double/debiased machine learning (DML) framework to reduce bias in two-stage IV regression.
result DML-IV outperforms state-of-the-art methods and learns high-performing policies.
GL-LowPopArt improves minimax-optimal estimation for trace regression.
problem Minimizing estimation error in generalized low-rank trace regression.
method Two-stage approach: nuclear norm regularization followed by matrix Catoni estimation.
result Achieves instance-wise optimal error bounds up to condition number.
We propose a two-stage hybrid approach with neural networks as the new feature construction algorithms for bankcard response classifications. The hybrid model uses a very simple neural network structure as the new feature construction tool in the first stage, then the newly created features are used as the additional i…
This paper proposes a general adaptive procedure for budget-limited predictor design in high dimensions called two-stage Sampling, Prediction and Adaptive Regression via Correlation Screening (SPARCS). SPARCS can be applied to high dimensional prediction problems in experimental science, medicine, finance, and engineer…
A novel framework for regression with multiple experts, addressing challenges in infinite and continuous label spaces.
problem Challenges in regression with multiple experts due to the infinite and continuous nature of the label space.
method Introduces a novel framework for regression with deferral, analyzing both single-stage and two-stage scenarios with new surrogate loss functions.
result Proves H-consistency bounds for both single-stage and two-stage methods, providing stronger guarantees than Bayes consistency. Kernel methods estimate causal effects with a single proxy for deterministic confounders.
problem Estimating causal effects with a single proxy for an unobserved confounder.
method Two kernel-based methods: two-stage regression and maximum moment restriction.
result Both kernel methods can consistently estimate the causal effect.
Method estimates treatment effects with continuous values, correcting for confounding.
problem Estimating treatment effects with continuous values, dealing with confounding.
method Two-stage kernel ridge regression: first stage learns response, second stage corrects for distribution shift.
result Optimal learning bounds achieved without estimating treatment density, adapts to unknown overlap and kernel spectral decay.
Debias concept-based explanations by removing confounding information.
problem Correlation between concepts and confounding features.
method Causal prior graph and two-stage regression technique.
result Success in removing biases and improving concept ranking.
We focus on the distribution regression problem: regressing to vector-valued outputs from probability measures. Many important machine learning and statistical tasks fit into this framework, including multi-instance learning and point estimation problems without analytical solution (such as hyperparameter or entropy es…
This thesis tackles learning with multi-class abstention and multi-expert deferral, improving model reliability and efficiency.
problem Improving model reliability and efficiency in large language models (LLMs) by leveraging multiple experts.
method Developed new surrogate losses and consistency guarantees for multi-class classification and regression with deferral.
result Strong consistency guarantees for surrogate losses in multi-class classification and regression with deferral.
Method estimates sparse inverse covariance and partial correlation matrices efficiently.
problem Sparse high-dimensional inverse covariance and partial correlation matrix estimation.
method Two-stage estimation method using partial regression with positive semi-definiteness.
result Efficient estimation of inverse covariance and partial correlation matrices with derived non-asymptotic rates.
Preprocessing data is an important step before any data analysis. In this paper, we focus on one particular aspect, namely scaling or normalization. We analyze various scaling methods in common use and study their effects on different statistical learning models. We will propose a new two-stage scaling method. First, w…
We show that the two-stage adaptive Lasso procedure (Zou, 2006) is consistent for high-dimensional model selection in linear and Gaussian graphical models. Our conditions for consistency cover more general situations than those accomplished in previous work: we prove that restricted eigenvalue conditions (Bickel et al.…
This paper explores the following question: what kind of statistical guarantees can be given when doing variable selection in high-dimensional models? In particular, we look at the error rates and power of some multi-stage regression methods. In the first stage we fit a set of candidate models. In the second stage we s…
Proposes KAR for nonlinear causal discovery using kernel methods.
problem Learning causal relationships in nonlinear settings.
method Kernel anchor regression (KAR) with improved three-stage nonparametric regression.
result KAR outperforms existing methods in nonlinear causal discovery.
Principal component regression (PCR) is a two-stage procedure that selects some principal components and then constructs a regression model regarding them as new explanatory variables. Note that the principal components are obtained from only explanatory variables and not considered with the response variable. To addre…
New adaptive models improve prediction accuracy with missing data.
problem Improving prediction accuracy with missing data entries.
method Adaptive optimization approach, learning imputation and regression simultaneously.
result 2-10% improvement in out-of-sample accuracy in strongly non-random missing data settings.
The paper analyzes methods for estimating linear functionals from observational data, proving upper bounds and showing optimal procedures.
problem Estimating linear functionals from observational data in causal inference and bandit literature.
method Two-stage procedures that first estimate treatment effect function, then use it to estimate the linear functional.
result Proves non-asymptotic upper bounds on mean-squared error for two-stage procedures and shows instance-dependent optimality.
We study the problem of variable selection in convex nonparametric regression. Under the assumption that the true regression function is convex and sparse, we develop a screening procedure to select a subset of variables that contains the relevant variables. Our approach is a two-stage quadratic programming method that…
Study semi-supervised learning with noisy proxy covariates, deriving bounds and showing gains.
problem Learning from noisy proxy covariates with scarce labels.
method Two-stage estimator learning kernel eigenfeatures from all proxy covariates and fitting a ridge predictor on labeled data.
result Finite sample bounds show fast labeled sample rates and consistent gains over supervised and semi-supervised baselines.
DML-CMR estimator reduces bias in CMR problems using deep neural networks.
problem Solving conditional moment restrictions with deep neural networks.
method Double/debiased machine learning framework for unbiased estimation.
result Achieves minimax optimal convergence rate of O(N−1/2). Proposes FARM model combining latent factor and sparse regression.
problem Testing adequacy of latent factor and sparse regression models.
method Factor Augmented sparse linear Regression Model (FARM) with FabTest and ANOVA type tests.
result Model robustness and effectiveness validated through experiments.
Instrumental variable (IV) regression is a strategy for learning causal relationships in observational data. If measurements of input X and output Y are confounded, the causal relationship can nonetheless be identified if an instrumental variable Z is available that influences X directly, but is conditionally independe…
Transformers can handle endogeneity in linear regression using IV methods.
problem Endogeneity in in-context linear regression models.
method Transformer architecture with gradient-based bi-level optimization and in-context pretraining.
result Transformers provide more robust predictions and estimates than 2SLS in endogenous scenarios.
Proposes a new network for accurate predictions and uncertainty estimation.
problem Uncertainty estimation in regression predictions without sacrificing accuracy.
method Decoupled two-stage training process with custom loss function.
result Reduces prediction error by 23-34% while maintaining 95% PICP.
We analyze the consistency of decision trees and random forests in regression.
problem Consistency of decision trees and random forests in regression.
method Elementary proofs following classical arguments for smoothing methods.
result Establish weak and almost sure convergence of honest trees and forest averages to the true regression function.
Proposes SHORE model for efficient MOR with sparsity and scalability.
problem Challenges of interpretability and scalability in MOR with high-dimensional outputs.
method Incorporates sparsity requirements and a two-stage optimization framework for efficient compression.
result Theoretical and empirical validation of the proposed framework's efficiency and accuracy.
We introduce a framework for performing regression between two Hilbert spaces. This is done based on Kirszbraun's extension theorem, to the best of our knowledge, the first application of this technique to supervised learning. We analyze the statistical and computational aspects of this method. We decompose this task i…
New method tackles endogeneity in online learning with improved regret bounds.
problem Endogeneity in real data due to omitted variables, strategic behaviors, etc.
method O2SLS (Online Two-Stage Least Squares) for Instrumental Variable (IV) regression.
result O2SLS achieves identification and oracle regret bounds for stochastic online learning.