Theoretical analysis shows pretext-based self-supervised learning can be boosted by downstream data under certain conditions.
problem Theoretical analysis of pretext-based self-supervised learning and downstream data refinement.
method Theoretical analysis and experiments on synthetic and real-world datasets.
result Theoretical lower bounds and experiments show that downstream data refinement can boost or hurt performance depending on conditions.
Paper defines and solves a problem in representation learning to ensure fairness with high confidence.
problem Learning fair representations with high confidence guarantees for all downstream tasks.
method Formally defines the problem, introduces FRG framework, proves high probability fairness, and demonstrates effectiveness empirically.
result FRG framework provides high-confidence guarantees for limiting unfairness across all downstream models and tasks.
Self-supervised metric learning boosts downstream tasks in multi-view data.
problem Improving distance-based downstream tasks without labeled data.
method Developed a statistical framework to study self-supervised metric learning in multi-view data.
result Self-supervised metric learning improves target distances for various downstream tasks.
This study examines how the size and alignment of pretraining data affect the performance of large language models on downstream tasks.
problem Understanding how the size and alignment of pretraining data impact the performance of large language models on downstream tasks.
method Investigated the scaling behavior of large language models in a transfer learning setting, focusing on machine translation tasks.
result The size of the finetuning dataset and the distribution alignment between pretraining and downstream data significantly influence the scaling behavior of downstream performance.
Study shows how pretraining robustness transfers to downstream tasks.
problem Understanding how robustness is transferred from pretraining to downstream tasks.
method Theoretical analysis and practical validation of robustness constraints.
result Robustness of a linear predictor on downstream tasks can be constrained by the robustness of its underlying representation.
End-to-end approach for weak supervision improves downstream model performance.
problem Data-labeling bottleneck in machine learning applications.
method Directly learning the downstream model by maximizing its agreement with probabilistic labels generated from weak supervision sources.
result Improved performance over prior work in terms of downstream model performance and robustness.
This paper characterizes VAE training pathologies and their effects on tasks.
problem Characterizing VAE training pathologies and their impact on downstream tasks.
method Concretely characterizing conditions for VAE training pathologies and their connection to specific downstream tasks.
result Connects VAE training pathologies to specific downstream tasks like learning compressed and disentangled representations, adversarial robustness, and semi-supervised learning.
New framework minimizes model complexity for improved few-shot learning.
problem Empirical benefits of pre-training scale with data size but lack theoretical explanation.
method Complexity Minimization framework for meta-representation learning.
result Theoretical analysis shows error rate improves with more meta-training data.
Mask-reconstruction pretraining helps in downstream tasks by capturing more semantic features.
problem How mask-reconstruction pretraining helps in downstream tasks and why it surpasses supervised learning.
method Theoretical analysis and experimental validation of mask-reconstruction pretraining (MRP) on auto-encoders.
result MRP provably captures more semantic features than supervised learning, leading to better performance in downstream tasks.
This work optimizes alignment and uniformity of features on a hypersphere for better downstream performance.
problem Improving the performance of contrastive representation learning.
method Identifying and optimizing alignment and uniformity of features on a hypersphere.
result Directly optimizing alignment and uniformity leads to comparable or better performance than contrastive learning.
Enhances performance on downstream tasks using multi-domain data.
problem Improving performance on tasks with limited downstream data.
method Deep transfer learning framework leveraging shared and domain-specific features.
result Significantly improves convergence rate for learning Lipschitz functions.
This work proposes a new pre-processing method for supervised learning to improve fairness without sacrificing utility.
problem Improving fairness in supervised learning without compromising model performance.
method Task-tailored pre-processing approach that balances fairness and utility.
result The proposed method preserves consistent trade-offs among multiple downstream models and improves fairness in computer vision tasks.
New framework explains how larger pre-trained models reduce downstream learning sample complexity.
problem Understanding why larger pre-trained models reduce sample complexity in downstream tasks.
method Introducing a novel framework called Caulking inspired by PEFT methods.
result Improved pre-trained models provably decrease downstream task sample complexity.
Optimal feature transfer identified through bias-variance analysis.
problem Optimizing feature transfer in transfer learning.
method Simple linear model with fine-grained bias-variance decomposition.
result Optimal pretrained feature transform is naturally sparse.
Minimal variations guide unsupervised learning for better downstream tasks.
problem Efficiently describing raw data for various future tasks.
method Minimal variations as a guiding principle for unsupervised representation learning.
result Unveiling minimal variations as a principle behind unsupervised learning.
New insights into contrastive learning reveal how projectors affect downstream performance.
problem Understanding how projectors in contrastive learning impact downstream linear classification accuracy.
method Identified and modeled two effects: expansion and shrinkage induced by contrastive loss.
result Linear projectors operating in the shrinkage regime hinder downstream classification accuracy.
FWC creates fair synthetic samples for machine learning tasks.
problem Addressing biases in machine learning models for fair decision-making.
method FWC uses an efficient majority minimization algorithm to minimize Wasserstein distance while enforcing demographic parity.
result FWC achieves a competitive fairness-utility tradeoff and reduces biases in predictions from large language models.
Optimizes ad pruning in sponsored search systems using reinforcement learning.
problem How to efficiently select top K ads from N candidates to maximize revenue.
method Model-free reinforcement learning approach considering downstream as a black-box environment.
result Remarkable improvements in revenue achieved through reinforcement learning.
Study shows scaling up models doesn't always improve downstream tasks.
problem Understanding why scaling up models doesn't always improve downstream performance.
method Systematic study of 4800 experiments on various models, analyzing performance on 20 downstream tasks.
result Performance on downstream tasks saturates as model size increases, revealing a nonlinear relationship.
Analysis of pretrained models' effectiveness in downstream tasks.
problem Understanding why pretrained models perform well in NLP tasks.
method Analyzed head and prompt tuning approaches using latent variable models.
result Prompt tuning provides stronger guarantees than head tuning.
New framework analyzes why more negative samples improve self-supervised learning performance.
problem Inconsistency between theoretical degradation and empirical improvement of downstream supervised tasks with more negative samples.
method Coupon collector's problem framework to analyze self-supervised representation learning with more negative samples.
result Bound can implicitly incorporate supervised loss in self-supervised loss by increasing negative samples.
This work investigates how neural collapse improves transfer learning for large-scale models.
problem Improving transfer learning for large-scale models with limited labeled data.
method Investigates neural collapse and develops a fine-tuning method using skip-connections.
result Feature collapse on downstream data correlates with higher transfer accuracy.
Object-centric learning improves generalization and robustness in multi-object scenes.
problem Improving generalization and robustness in neural networks for scenes with multiple objects.
method Training state-of-the-art unsupervised models on multi-object datasets and evaluating segmentation metrics and downstream tasks.
result Object-centric representations are useful for downstream tasks and generally robust to most distribution shifts affecting objects, but less so for less structured shifts.
Contrastive learning performance doesn't degrade with more negative samples.
problem Theoretical and empirical evidence of negative samples hurting performance in contrastive learning.
method Simple theoretical setting and empirical support on CIFAR-10 and CIFAR-100 datasets.
result Contrastive learning performance does not degrade with the number of negative samples.
Self-supervised learning improves few-shot classification and segmentation on point clouds.
problem Efficiently learn from limited labeled data in point cloud applications.
method Hierarchical cover-tree partitioning for self-supervised pre-training; restricted to support set for few-shot learning.
result Self-supervised learning significantly improves downstream classification and segmentation accuracy.
This paper evaluates fairness in deep metric learning and proposes a method to reduce subgroup performance gaps.
problem The negative impact of deep metric learning representations on minority subgroup performance in downstream tasks.
method Definition of fairness in DML through inter-class, intra-class, and uniformity properties; finDML benchmark; Partial Attribute De-correlation (PARADE) method.
result Bias in DML representations propagates to downstream tasks, even with balanced training data.
Theoretical study shows adversarial training improves robustness in deep learning models.
problem Ensuring robustness in pre-trained deep learning models.
method Theoretical analysis of adversarial training and feature purification in two-layer neural networks.
result Adversarial training leads to feature purification, making models more robust to attacks.
TaskMet learns a metric to improve model performance on unseen tasks.
problem Deep models trained on one task may struggle on another task due to conflicting objectives.
method TaskMet learns a metric in the prediction space to balance task and prediction losses.
result TaskMet achieves better performance on downstream tasks without altering the prediction model.
New method predicts wind farm power and wakes using weather patterns.
problem Inefficient and computationally intensive wind energy resource assessment.
method Unsupervised clustering of ERA5 data on wind velocity, WRF simulations at cluster centers, and post-processing.
result Accurate long-term predictions of power and wakes with reduced computational time.
Proposes a new metric to quantify the difference between neural network representations based on downstream task performance.
problem The lack of a consistent metric to measure the difference between neural network representations.
method Introduced the Transferred Discrepancy (TD) metric, which evaluates the difference between representations based on their performance on downstream tasks.
result TD provides fine-grained information for various downstream tasks and can evaluate the effectiveness of different training strategies.
PEARL combines multiple representation learning methods to enhance model performance.
problem Different representation learning methods extract distinct data aspects, potentially missing important insights.
method Combines multiple representation learning approaches using surrogate loss functions for efficient weight estimation.
result Asymptotically achieves optimal performance in downstream tasks, assigning nonzero weights to correctly specified models.
Contrastive learning outperforms autoencoders and GANs in feature recovery and downstream tasks.
problem Theoretical understanding of contrastive learning's superiority in feature learning.
method Theoretical analysis of contrastive learning in linear representation settings.
result Contrastive learning outperforms autoencoders and GANs for feature recovery and in-domain downstream tasks.
Identifies optimal base features for zero-shot adaptation in reinforcement learning.
problem Unclear what constitutes a good set of base features for a wide range of downstream tasks.
method Identifies optimal base features based on downstream performance, without assuming downstream tasks are linear.
result Optimal base features are the same across three task families, differing from Laplacian eigenfunctions.
Improved VAE representations lead to better image classification.
problem VAE representations are inferior to non-latent models for image classification.
method Used a decoder that prefers local features, improving global feature capture in latent variables.
result Significant improvement in downstream semantic classification tasks.
Study shows uncertainty of deep learning models can be measured from their embeddings.
problem Uncertainty in contrastive learning models for critical applications.
method Estimating the distribution of training data in embedding space and accounting for local consistency.
result Uncertainty of an embedding vector correlates strongly with downstream accuracy.
NCL improves interpretability of deep features by enforcing non-negativity.
problem Lack of interpretability in deep representations.
method Non-negative Contrastive Learning (NCL) using non-negativity constraints.
result NCL outperforms standard contrastive learning in feature disentanglement and selection.
Framework fine-tunes foundation models with semi-supervised learning for downstream tasks and latent spaces.
problem Training foundation models with limited labelled data.
method Mutual information decomposition for downstream and latent spaces, semi-supervised fine-tuning.
result Significant improvements in classification tasks under low-labelled conditions.
New algorithm REFUEL shows multitask representation learning is more sample-efficient in RL.
problem Understanding the benefit of representation learning in reinforcement learning.
method Developed REFUEL algorithm for multitask low-rank RL, analyzing both upstream and downstream tasks.
result Multitask representation learning is provably more sample-efficient than individual task learning.
The paper explores how word embeddings affect the stability of downstream NLP models.
problem Small changes in training data can cause significant changes in model predictions.
method Empirical and theoretical analysis of embedding instability, including the introduction of eigenspace instability measure.
result Increasing embedding memory can reduce the disagreement in predictions by 5% to 37%.
New bounds for contrastive learning handle domain shifts and generalization.
problem Domain shifts and generalization challenges in downstream tasks.
method Novel generalization bounds accounting for both domain shift and generalization.
result Performance of contrastively learned representations depends on statistical discrepancy between pretraining and downstream distributions.
New method optimizes treatment policies to avoid winner's curse.
problem Winner's curse in treatment policy optimization.
method Inference-aware policy optimization.
result Optimizes for both estimated performance and downstream evaluation.
New theory explains contrastive learning via overlapping augmented views.
problem Lack of theoretical understanding of contrastive learning.
method Augmentation overlap perspective to improve downstream performance.
result Asymptotically closed bounds for downstream performance under weaker assumptions.
This paper proves the theoretical advantage of unsupervised pretraining for machine learning tasks.
problem Understanding why unsupervised pretraining helps in machine learning tasks.
method A generic framework using Maximum Likelihood Estimation (MLE) for unsupervised pretraining and Empirical Risk Minimization (ERM) for downstream tasks.
result Proves an excess risk of i l d e O ( C Φ / m + C Ψ / n ) ilde{\mathcal{O}}(\sqrt{\mathcal{C}_Φ/m} + \sqrt{\mathcal{C}_Ψ/n}) i l d e O ( C Φ / m + C Ψ / n ) for downstream tasks under mild conditions. A central goal of unsupervised learning is to acquire representations from unlabeled data or experience that can be used for more effective learning of downstream tasks from modest amounts of labeled data. Many prior unsupervised learning works aim to do so by developing proxy objectives based on reconstruction, disent…
Develops a statistical framework for self-supervised representation learning using data augmentation.
problem Lack of theoretical understanding of data augmentation in nonlinear settings.
method Augmentation invariant manifold learning framework and stochastic optimization algorithm.
result Improves downstream analysis by exploiting manifold's geometric structure and invariant property of augmented data.
Fine-tunes GNNs by preserving generative patterns to improve transferability.
problem Vanilla fine-tuning fails due to structural divergence between pre-training and downstream graphs.
method G-Tuning, which reconstructs the generative patterns of the downstream graph using graphon bases.
result G-Tuning achieves an average improvement of 0.5% and 2.6% on in-domain and out-of-domain transfer learning experiments.
Proposes FairRR to improve fairness in machine learning models through randomized response.
problem Achieving group fairness in machine learning models.
method Formulates group fairness as optimizing a design matrix in Randomized Response, proposing FairRR.
result Demonstrates FairRR yields excellent model utility and fairness.
Contrastive learning harms minority group representations, affecting downstream tasks.
problem Representation harm in contrastive learning, especially affecting minority groups.
method Causal mediation analysis and stochastic block model explanation.
result Representation harm in contrastive learning is partly responsible for allocation harm in downstream tasks.