The study examines when to trust confidence thresholding in pseudo-labelling regression.
problem Calibrated probabilities from classifiers used for pseudo-labelling need careful handling to avoid bias in downstream regression.
method Developed a diagnostic apparatus to predict and bound the bias induced by confidence thresholding, derived a closed-form expression for the attenuation bias.
result The bias can be predicted from the residual score variance V∗, motivating a structural separation between classifier features and downstream controls. A new method for decision-focused learning reduces computational cost.
problem Efficiently solving combinatorial problems with uncertain parameters.
method Reframed as cost-sensitive multi-output regression, with novel loss components.
result Comparable downstream task quality with reduced computational cost.
New framework minimizes model complexity for improved few-shot learning.
problem Empirical benefits of pre-training scale with data size but lack theoretical explanation.
method Complexity Minimization framework for meta-representation learning.
result Theoretical analysis shows error rate improves with more meta-training data.
Generative Adversarial Regression (GAR) learns risk scenarios robustly across policies.
problem Learning risk scenarios for conditional risk objectives.
method Generative adversarial framework for risk matching.
result GAR produces more stable and risk-preserving scenarios than baselines.
Enhances performance on downstream tasks using multi-domain data.
problem Improving performance on tasks with limited downstream data.
method Deep transfer learning framework leveraging shared and domain-specific features.
result Significantly improves convergence rate for learning Lipschitz functions.
Training a source model optimally for its own task is suboptimal for downstream transfer.
problem The optimality of a source model for its own task hinders downstream transfer performance.
method Analyzes L2-SP ridge regression, characterizes transfer-optimal source penalty, and identifies alignment-dependent effects.
result Transfer benefits from stronger source regularization when aligned imperfectly, and from weaker regularization when aligned perfectly.
CIT and CIF improve feature selection for downstream prediction.
problem Feature selection bias in machine learning models.
method Conditional inference trees and forests with Bonferroni correction.
result CIF ranks top 3 among 18 regression methods and top 4 among 17 classification methods.
This work proposes a new pre-processing method for supervised learning to improve fairness without sacrificing utility.
problem Improving fairness in supervised learning without compromising model performance.
method Task-tailored pre-processing approach that balances fairness and utility.
result The proposed method preserves consistent trade-offs among multiple downstream models and improves fairness in computer vision tasks.
Develops UKP for comparing feature representations in multitask learning.
problem Comparing feature representations learned by different models without access to test data.
method Uniform Kernel Prober (UKP) for comparing representations in kernel ridge regression tasks.
result UKP provides a uniform measure of prediction error on test data without access to test data.
This work analyzes the role of data augmentation in self-supervised learning using RKHS approximation and regression.
problem Limited theoretical understanding of the role of data augmentation in self-supervised learning.
method Geometric characterization of the target function given by augmentation, proving generalization bounds.
result Two generalization bounds are derived, one free of model complexity, the other specific to near-optimal encoders.
New adaptive models improve prediction accuracy with missing data.
problem Improving prediction accuracy with missing data entries.
method Adaptive optimization approach, learning imputation and regression simultaneously.
result 2-10% improvement in out-of-sample accuracy in strongly non-random missing data settings.
Compressing word embeddings is important for deploying NLP models in memory-constrained settings. However, understanding what makes compressed embeddings perform well on downstream tasks is challenging---existing measures of compression quality often fail to distinguish between embeddings that perform well and those th…
New statistical theory explains contrastive learning effectiveness.
problem Understanding why contrastive learning works well for representation extraction.
method Developed a new theoretical framework based on approximate sufficient statistics.
result Near-sufficient encoders derived from contrastive learning can be adapted for downstream tasks.
New method uses imperfect LLM annotations for valid statistical inference in social science.
problem Inaccurate large language model annotations in social science research.
method Design-based supervised learning (DSL) combining imperfect LLM surrogates with gold-standard labels.
result DSL provides valid statistical inference with comparable predictive accuracy to existing methods.
New theory explains when pre-trained models can improve downstream tasks.
problem Lack of theoretical understanding of task similarity for transfer learning.
method Feature-centric viewpoint, theoretical results on transferability phase diagram.
result Transfer learning outperforms training from scratch when target task is well represented in feature space.
Generative models often fail to preserve joint structure despite matching marginals.
problem Generative models fail to capture complex dependencies beyond univariate marginals.
method Introduced D_Sigma(P,Q) = ||Sigma_P - Sigma_Q||_F to measure covariance-level dependence fidelity.
result Covariance-level divergence can lead to structural instability in downstream inference.
The study analyzes transfer learning in infinite-width neural networks, improving generalization on target tasks.
problem Improving generalization in neural networks when using pretraining on a source task.
method Developed a theory under gradient flow for infinitely wide networks, analyzing fine-tuning and joint pretraining.
result Summary statistics of randomly initialized networks after pretraining are adaptive kernels that depend on both source and target data.
Study highlights how model choice affects uncertainty estimation in neural network regression.
problem Uncertainty estimation under model misspecification in neural network regression.
method Analyzed the impact of model choice on uncertainty estimation in neural network regression, focusing on aleatoric and epistemic uncertainties.
result Model misspecification leads to unreliable uncertainty estimates, highlighting the importance of choosing appropriate models.
Proposes a method for differentially private linear regression and synthetic data generation.
problem Lack of valid inference and synthetic data generation methods for small-scale datasets in privacy-aware settings.
method Gaussian differentially private linear regression with bias-corrected estimator and SDG procedure.
result Improves accuracy and provides valid confidence intervals for downstream tasks.
BCDP enhances privacy by protecting sensitive features more precisely.
problem Uniform privacy protection in LDP degrades performance for sensitive features.
method Bayesian Coordinate Differential Privacy (BCDP) adjusts privacy protection per feature sensitivity.
result BCDP improves accuracy in downstream tasks without sacrificing privacy.
New methods for sketching non-PSD matrices improve regression and optimization tasks.
problem Efficiently handling non-PSD matrices in computations.
method Developed novel matrix sketching techniques for non-PSD and complex matrices.
result Improved performance in convex and non-convex optimization, regression, and vector-matrix-vector queries.
Unified framework for CVA sensitivities, hedging, and risk assessment.
problem Computing and managing Credit Value Adjustment (CVA) sensitivities and risks.
method Probabilistic machine learning and refined regression on simulated data, validated by Monte Carlo methods.
result Identification of optimal sensitivities for practical tasks like hedging and risk assessment.
Linear regression is a classic method of data analysis. In recent years, sketching -- a method of dimension reduction using random sampling, random projections, or both -- has gained popularity as an effective computational approximation when the number of observations greatly exceeds the number of variables. In this p…
Theoretical analysis shows pretext-based self-supervised learning can be boosted by downstream data under certain conditions.
problem Theoretical analysis of pretext-based self-supervised learning and downstream data refinement.
method Theoretical analysis and experiments on synthetic and real-world datasets.
result Theoretical lower bounds and experiments show that downstream data refinement can boost or hurt performance depending on conditions.
This study examines how the size and alignment of pretraining data affect the performance of large language models on downstream tasks.
problem Understanding how the size and alignment of pretraining data impact the performance of large language models on downstream tasks.
method Investigated the scaling behavior of large language models in a transfer learning setting, focusing on machine translation tasks.
result The size of the finetuning dataset and the distribution alignment between pretraining and downstream data significantly influence the scaling behavior of downstream performance.
Empirical error estimates improve graph sparsification reliability.
problem Uncertainty in sparsification error limits downstream computations reliability.
method Data-driven approach to compute empirical error estimates.
result Empirical error estimates provide theoretical guarantees and are computationally feasible.
Work proposes a new framework to improve uncertainty estimation in deep Bayesian models.
problem Traditional training procedures underestimate uncertainty in NLMs, leading to unreliable predictions.
method Introduces a novel training framework that captures useful predictive uncertainties for out-of-distribution inputs.
result Demonstrates that traditional methods for NLMs significantly underestimate uncertainty and propose a new framework to address this issue.
Paper defines and solves a problem in representation learning to ensure fairness with high confidence.
problem Learning fair representations with high confidence guarantees for all downstream tasks.
method Formally defines the problem, introduces FRG framework, proves high probability fairness, and demonstrates effectiveness empirically.
result FRG framework provides high-confidence guarantees for limiting unfairness across all downstream models and tasks.
Personalizes pre-trained models for nonparametric regression with limited data.
problem Improving data efficiency in nonparametric regression with few samples.
method Develops a theoretical framework and algorithms for few-shot personalization of black-box models.
result Achieves minimax optimal rate for personalization in nonparametric regression.
Study shows how pretraining robustness transfers to downstream tasks.
problem Understanding how robustness is transferred from pretraining to downstream tasks.
method Theoretical analysis and practical validation of robustness constraints.
result Robustness of a linear predictor on downstream tasks can be constrained by the robustness of its underlying representation.
This paper explores how model complexity affects performance in large language models.
problem Understanding how model complexity impacts performance in large language models.
method Employing Hidden Markov Models (HMMs) to model autoregressive LLMs and investigating the relationship between model complexity and generalization capability.
result Model complexity initially increases risk, then decreases, showcasing a 'double descent' phenomenon.
Self-supervised metric learning boosts downstream tasks in multi-view data.
problem Improving distance-based downstream tasks without labeled data.
method Developed a statistical framework to study self-supervised metric learning in multi-view data.
result Self-supervised metric learning improves target distances for various downstream tasks.
Study shows scaling up models doesn't always improve downstream tasks.
problem Understanding why scaling up models doesn't always improve downstream performance.
method Systematic study of 4800 experiments on various models, analyzing performance on 20 downstream tasks.
result Performance on downstream tasks saturates as model size increases, revealing a nonlinear relationship.
Analysis of pretrained models' effectiveness in downstream tasks.
problem Understanding why pretrained models perform well in NLP tasks.
method Analyzed head and prompt tuning approaches using latent variable models.
result Prompt tuning provides stronger guarantees than head tuning.
Optimal convex loss function improves regression coefficient estimation.
problem Asymptotic variance improvement in linear regression estimation.
method Score matching extension for log-concave projection.
result Semiparametric estimator attains minimal asymptotic covariance.
Graph representation learning is to learn universal node representations that preserve both node attributes and structural information. The derived node representations can be used to serve various downstream tasks, such as node classification and node clustering. When a graph is heterogeneous, the problem becomes more…
This paper characterizes VAE training pathologies and their effects on tasks.
problem Characterizing VAE training pathologies and their impact on downstream tasks.
method Concretely characterizing conditions for VAE training pathologies and their connection to specific downstream tasks.
result Connects VAE training pathologies to specific downstream tasks like learning compressed and disentangled representations, adversarial robustness, and semi-supervised learning.
New method improves Gaussian kernel approximations for high-frequency data.
problem Limited scalability of kernel-based models to large data sets.
method Local random feature approximations using Maclaurin expansions and polynomial sketches.
result Significant improvement in kernel approximations and downstream performance for high-frequency data.
Popular graph neural networks implement convolution operations on graphs based on polynomial spectral filters. In this paper, we propose a novel graph convolutional layer inspired by the auto-regressive moving average (ARMA) filter that, compared to polynomial ones, provides a more flexible frequency response, is more …
The paper proposes a method to assess survey data credibility without needing many samples, regardless of data dimension.
problem Assessing the credibility of survey data across different dimensions.
method Task-based approach and model-specific distance metric for verifying survey data credibility in regression models.
result The sample complexity of the proposed algorithm is independent of the data dimension, making it more efficient.
New framework explains how larger pre-trained models reduce downstream learning sample complexity.
problem Understanding why larger pre-trained models reduce sample complexity in downstream tasks.
method Introducing a novel framework called Caulking inspired by PEFT methods.
result Improved pre-trained models provably decrease downstream task sample complexity.
Noise-aware DP inference improves accuracy for complex models.
problem Inaccurate results and biases in DP inference for complex models.
method Noise-aware stochastic gradient variational inference.
result Accurate coverages and predictive probabilities for complex models.
CoLoRA leverages task similarity to boost fine-tuning efficiency.
problem Efficiently fine-tuning large foundation models with scarce labeled data.
method CoLoRA trains a shared adapter for task similarity and personalized adapters for user-specific tasks.
result CoLoRA significantly boosts fine-tuning performance when tasks are similar.
End-to-end approach for weak supervision improves downstream model performance.
problem Data-labeling bottleneck in machine learning applications.
method Directly learning the downstream model by maximizing its agreement with probabilistic labels generated from weak supervision sources.
result Improved performance over prior work in terms of downstream model performance and robustness.
UBM transfers bias mitigation from upstream to downstream tasks efficiently.
problem Bias in fine-tuned language models across various tasks.
method Apply bias mitigation to an upstream model, then fine-tune a downstream model on this mitigated model.
result UBM effects transfer to new downstream tasks, creating less biased models.
Mask-reconstruction pretraining helps in downstream tasks by capturing more semantic features.
problem How mask-reconstruction pretraining helps in downstream tasks and why it surpasses supervised learning.
method Theoretical analysis and experimental validation of mask-reconstruction pretraining (MRP) on auto-encoders.
result MRP provably captures more semantic features than supervised learning, leading to better performance in downstream tasks.
Paper proposes a new method to learn distribution kernels via entropy maximization.
problem Challenges in applying kernel methods to distribution regression tasks.
method Proposes a novel objective for unsupervised learning of data-dependent distribution kernels based on entropy maximization.
result Demonstrates the effectiveness of the learned kernel across different modalities.
Develops efficient inference for noise heterogeneity in machine learning models.
problem Downstream procedures based on residuals can be biased in additive noise models.
method Semiparametrically efficient inference using a novel Hilbert-valued one-step estimator.
result Constructs tests and confidence intervals for residual independence and goodness of fit.