CARV reduces compute cost for downstream pipelines using diffusion models.
problem High variance in Monte Carlo estimators from diffusion models limits compute efficiency.
method CARV uses hierarchical MC estimation with amortized upstream computation and stratified-inverse-CDF.
result CARV delivers 2-3x effective compute multipliers without changing the objective.
The paper predicts loss scaling across different datasets and compute scales.
problem Predicting loss scaling across different datasets and compute scales.
method Derive shifted power law relationships between train and test losses.
result Shifted power law relationships hold for various datasets and tasks, improving prediction accuracy.
Optimizes ad pruning in sponsored search systems using reinforcement learning.
problem How to efficiently select top K ads from N candidates to maximize revenue.
method Model-free reinforcement learning approach considering downstream as a black-box environment.
result Remarkable improvements in revenue achieved through reinforcement learning.
This work proposes a new pre-processing method for supervised learning to improve fairness without sacrificing utility.
problem Improving fairness in supervised learning without compromising model performance.
method Task-tailored pre-processing approach that balances fairness and utility.
result The proposed method preserves consistent trade-offs among multiple downstream models and improves fairness in computer vision tasks.
Empirical error estimates improve graph sparsification reliability.
problem Uncertainty in sparsification error limits downstream computations reliability.
method Data-driven approach to compute empirical error estimates.
result Empirical error estimates provide theoretical guarantees and are computationally feasible.
New method predicts wind farm power and wakes using weather patterns.
problem Inefficient and computationally intensive wind energy resource assessment.
method Unsupervised clustering of ERA5 data on wind velocity, WRF simulations at cluster centers, and post-processing.
result Accurate long-term predictions of power and wakes with reduced computational time.
A new method for experimental design focuses on predicting downstream quantities of interest.
problem Designs that maximize parameter learning may not maximize downstream quantity prediction.
method Likelihood-free goal-oriented optimal experimental design (LF-GO-OED) using ABC density ratio estimation.
result LF-GO-OED maximizes the expected information gain for downstream quantities.
This work investigates how neural collapse improves transfer learning for large-scale models.
problem Improving transfer learning for large-scale models with limited labeled data.
method Investigates neural collapse and develops a fine-tuning method using skip-connections.
result Feature collapse on downstream data correlates with higher transfer accuracy.
The study examines when to trust confidence thresholding in pseudo-labelling regression.
problem Calibrated probabilities from classifiers used for pseudo-labelling need careful handling to avoid bias in downstream regression.
method Developed a diagnostic apparatus to predict and bound the bias induced by confidence thresholding, derived a closed-form expression for the attenuation bias.
result The bias can be predicted from the residual score variance V∗, motivating a structural separation between classifier features and downstream controls. PEARL combines multiple representation learning methods to enhance model performance.
problem Different representation learning methods extract distinct data aspects, potentially missing important insights.
method Combines multiple representation learning approaches using surrogate loss functions for efficient weight estimation.
result Asymptotically achieves optimal performance in downstream tasks, assigning nonzero weights to correctly specified models.
Theoretical analysis shows pretext-based self-supervised learning can be boosted by downstream data under certain conditions.
problem Theoretical analysis of pretext-based self-supervised learning and downstream data refinement.
method Theoretical analysis and experiments on synthetic and real-world datasets.
result Theoretical lower bounds and experiments show that downstream data refinement can boost or hurt performance depending on conditions.
This study examines how the size and alignment of pretraining data affect the performance of large language models on downstream tasks.
problem Understanding how the size and alignment of pretraining data impact the performance of large language models on downstream tasks.
method Investigated the scaling behavior of large language models in a transfer learning setting, focusing on machine translation tasks.
result The size of the finetuning dataset and the distribution alignment between pretraining and downstream data significantly influence the scaling behavior of downstream performance.
New method uses imperfect LLM annotations for valid statistical inference in social science.
problem Inaccurate large language model annotations in social science research.
method Design-based supervised learning (DSL) combining imperfect LLM surrogates with gold-standard labels.
result DSL provides valid statistical inference with comparable predictive accuracy to existing methods.
Paper defines and solves a problem in representation learning to ensure fairness with high confidence.
problem Learning fair representations with high confidence guarantees for all downstream tasks.
method Formally defines the problem, introduces FRG framework, proves high probability fairness, and demonstrates effectiveness empirically.
result FRG framework provides high-confidence guarantees for limiting unfairness across all downstream models and tasks.
Study shows how pretraining robustness transfers to downstream tasks.
problem Understanding how robustness is transferred from pretraining to downstream tasks.
method Theoretical analysis and practical validation of robustness constraints.
result Robustness of a linear predictor on downstream tasks can be constrained by the robustness of its underlying representation.
New theory explains contrastive learning via overlapping augmented views.
problem Lack of theoretical understanding of contrastive learning.
method Augmentation overlap perspective to improve downstream performance.
result Asymptotically closed bounds for downstream performance under weaker assumptions.
Self-supervised metric learning boosts downstream tasks in multi-view data.
problem Improving distance-based downstream tasks without labeled data.
method Developed a statistical framework to study self-supervised metric learning in multi-view data.
result Self-supervised metric learning improves target distances for various downstream tasks.
Study shows scaling up models doesn't always improve downstream tasks.
problem Understanding why scaling up models doesn't always improve downstream performance.
method Systematic study of 4800 experiments on various models, analyzing performance on 20 downstream tasks.
result Performance on downstream tasks saturates as model size increases, revealing a nonlinear relationship.
Analysis of pretrained models' effectiveness in downstream tasks.
problem Understanding why pretrained models perform well in NLP tasks.
method Analyzed head and prompt tuning approaches using latent variable models.
result Prompt tuning provides stronger guarantees than head tuning.
A new method for decision-focused learning reduces computational cost.
problem Efficiently solving combinatorial problems with uncertain parameters.
method Reframed as cost-sensitive multi-output regression, with novel loss components.
result Comparable downstream task quality with reduced computational cost.
This paper characterizes VAE training pathologies and their effects on tasks.
problem Characterizing VAE training pathologies and their impact on downstream tasks.
method Concretely characterizing conditions for VAE training pathologies and their connection to specific downstream tasks.
result Connects VAE training pathologies to specific downstream tasks like learning compressed and disentangled representations, adversarial robustness, and semi-supervised learning.
Generative models often fail to preserve joint structure despite matching marginals.
problem Generative models fail to capture complex dependencies beyond univariate marginals.
method Introduced D_Sigma(P,Q) = ||Sigma_P - Sigma_Q||_F to measure covariance-level dependence fidelity.
result Covariance-level divergence can lead to structural instability in downstream inference.
Proposes a new model for joint probability distributions in computer vision.
problem Limitation of existing models in meeting diverse downstream tasks.
method Uses parametric conditional probability distributions for each group of variables conditioned on the rest.
result Models can be used for any downstream task without task-specific design.
New framework explains how larger pre-trained models reduce downstream learning sample complexity.
problem Understanding why larger pre-trained models reduce sample complexity in downstream tasks.
method Introducing a novel framework called Caulking inspired by PEFT methods.
result Improved pre-trained models provably decrease downstream task sample complexity.
Adversarially trained transformers can learn robustly across tasks with minimal tuning.
problem Adversarial attacks and the high cost of adversarial training.
method Adversarial pretraining followed by in-context learning.
result Single-layer linear transformers can generalize robustly to unseen tasks.
End-to-end approach for weak supervision improves downstream model performance.
problem Data-labeling bottleneck in machine learning applications.
method Directly learning the downstream model by maximizing its agreement with probabilistic labels generated from weak supervision sources.
result Improved performance over prior work in terms of downstream model performance and robustness.
DHOG improves unsupervised clustering accuracy on image benchmarks.
problem Local optima in mutual information maximisation lead to suboptimal representations.
method Deep hierarchical object grouping (DHOG) computes multiple discrete representations in a hierarchical order.
result DHOG achieves new state-of-the-art results on three main benchmarks.
Representation learning on graphs has emerged as a powerful mechanism to automate feature vector generation for downstream machine learning tasks. The advances in representation on graphs have centered on both homogeneous and heterogeneous graphs, where the latter presenting the challenges associated with multi-typed n…
LITE efficiently estimates Gaussian PoM with linear time and memory complexity.
problem Estimating the probability of maximality (PoM) of Gaussian vectors efficiently.
method LITE: entropy-regularized UCB approach for almost-linear time and memory complexity.
result Achieves state-of-the-art accuracy with significantly faster performance than existing methods.
UBM transfers bias mitigation from upstream to downstream tasks efficiently.
problem Bias in fine-tuned language models across various tasks.
method Apply bias mitigation to an upstream model, then fine-tune a downstream model on this mitigated model.
result UBM effects transfer to new downstream tasks, creating less biased models.
Mask-reconstruction pretraining helps in downstream tasks by capturing more semantic features.
problem How mask-reconstruction pretraining helps in downstream tasks and why it surpasses supervised learning.
method Theoretical analysis and experimental validation of mask-reconstruction pretraining (MRP) on auto-encoders.
result MRP provably captures more semantic features than supervised learning, leading to better performance in downstream tasks.
A new method for fair classification using characteristic function distance.
problem Fairness in high-stakes decision-making with sensitive groups.
method Proposes a novel approach based on characteristic function distance to ensure minimal sensitive information in learned representations.
result Consistently matches or achieves better fairness and predictive accuracy than existing methods.
Enhances performance on downstream tasks using multi-domain data.
problem Improving performance on tasks with limited downstream data.
method Deep transfer learning framework leveraging shared and domain-specific features.
result Significantly improves convergence rate for learning Lipschitz functions.
This work optimizes alignment and uniformity of features on a hypersphere for better downstream performance.
problem Improving the performance of contrastive representation learning.
method Identifying and optimizing alignment and uniformity of features on a hypersphere.
result Directly optimizing alignment and uniformity leads to comparable or better performance than contrastive learning.
ASCEND discovers causal relationships in multi-omics data by leveraging known hierarchical structure.
problem Causal inference in high-dimensional multi-omics data, especially when ignoring the hierarchical structure.
method Two-tiered divide-and-conquer strategy with ancestral conditioning sets.
result Achieves polynomial-time complexity and accurately recovers ancestral relationships.
The paper finds a surprising positive correlation between upstreamness and downstreamness in global value chains.
problem The puzzling positive correlation between upstreamness and downstreamness in industries and countries.
method Analysis of a simple model of random Input/Output tables and experiments on empirical data.
result Upstreamness and downstreamness of the same industrial sector/country are positively correlated with a slope close to +1.
Optimal feature transfer identified through bias-variance analysis.
problem Optimizing feature transfer in transfer learning.
method Simple linear model with fine-grained bias-variance decomposition.
result Optimal pretrained feature transform is naturally sparse.
A new metric FRD improves comparing medical images.
problem Comparing medical images for distribution or domain differences.
method Developed a new metric FRD using standardized radiomic features.
result FRD outperforms other metrics in various medical imaging applications.
HyperINF improves influence function estimation for large models with better accuracy and efficiency.
problem Inaccurate and computationally expensive influence function estimation for large-scale models.
method HyperINF leverages Schulz's iterative algorithm and GFIM for low-rank approximation of Hessian matrix.
result HyperINF achieves superior accuracy and performance compared to existing methods on LoRA-tuned models.
New findings show BERT subnetworks can train independently and transfer to various tasks.
problem Finding smaller subnetworks that can train independently and transfer to other tasks.
method Examined pre-trained BERT models for subnetworks that can train independently and transfer to various downstream tasks.
result Found subnetworks at 40% to 90% sparsity that can train independently and transfer to various tasks.
New framework minimizes model complexity for improved few-shot learning.
problem Empirical benefits of pre-training scale with data size but lack theoretical explanation.
method Complexity Minimization framework for meta-representation learning.
result Theoretical analysis shows error rate improves with more meta-training data.
This paper develops a bootstrap method to estimate errors in Random Fourier Features.
problem Inability to estimate the error of Random Fourier Features approximations.
method Develops a bootstrap approach to numerically estimate the errors of RFF approximations.
result Specific, flexible, and adaptive error estimates for RFF approximations.
Bayesian neural networks (BNNs) allow us to reason about uncertainty in a principled way. Stochastic Gradient Langevin Dynamics (SGLD) enables efficient BNN learning by drawing samples from the BNN posterior using mini-batches. However, SGLD and its extensions require storage of many copies of the model parameters, a p…
FWC creates fair synthetic samples for machine learning tasks.
problem Addressing biases in machine learning models for fair decision-making.
method FWC uses an efficient majority minimization algorithm to minimize Wasserstein distance while enforcing demographic parity.
result FWC achieves a competitive fairness-utility tradeoff and reduces biases in predictions from large language models.
New insights into contrastive learning reveal how projectors affect downstream performance.
problem Understanding how projectors in contrastive learning impact downstream linear classification accuracy.
method Identified and modeled two effects: expansion and shrinkage induced by contrastive loss.
result Linear projectors operating in the shrinkage regime hinder downstream classification accuracy.
Study questions the reliability of uncertainty quantification in evidential deep learning.
problem Reliability of uncertainty quantification in evidential deep learning.
method Analysis of evidential deep learning methods, revealing their limitations and interpreting them as out-of-distribution detection algorithms.
result EDL methods are unreliable in quantifying uncertainty, even when effective on downstream tasks.
New method learns robust meta-representations for fast task adaptation.
problem Transferability of meta-representations in fast adaptation.
method Decoupled encoder-decoder approach with contrastive objective.
result Meta-representations improve downstream performance and noise robustness.
Proposes a multi-task learning model using variational information bottleneck.
problem Balancing performance and robustness across different tasks in multi-task learning.
method Variational Information Bottleneck (VIB) architecture for multi-task learning.
result The proposed model achieves competitive prediction accuracy under adversarial attacks.