Study shows how pretraining robustness transfers to downstream tasks.
problem Understanding how robustness is transferred from pretraining to downstream tasks.
method Theoretical analysis and practical validation of robustness constraints.
result Robustness of a linear predictor on downstream tasks can be constrained by the robustness of its underlying representation.
New method optimizes experimental design for specific applications.
problem Inability to adapt causal inference methods to specific downstream applications.
method Task-specific experimental design and sampling strategies.
result Significantly reduces data requirements for achieving RCT performance.
End-to-end approach for weak supervision improves downstream model performance.
problem Data-labeling bottleneck in machine learning applications.
method Directly learning the downstream model by maximizing its agreement with probabilistic labels generated from weak supervision sources.
result Improved performance over prior work in terms of downstream model performance and robustness.
This work studies the impact of intra-/inter-class diversity on pre-training datasets and finds a balance for optimal performance.
problem The impact of intra-/inter-class diversity on supervised pre-training datasets and their effect on downstream tasks.
method Empirical study and theoretical analysis of the relationship between diversity types and downstream performance.
result The optimal class-to-sample ratio is invariant to the size of the pre-training dataset and can be predicted.
Paper examines adversarial attacks on weather forecasting models, focusing on TC trajectory prediction.
problem Adversarial attacks can mislead downstream TC trajectory predictions in DLWF models.
method Proposes Cyc-Attack, a method using a surrogate model and skewness-aware loss function to generate adversarial TC paths.
result Cyc-Attack achieves higher true positive rates and lower false alarm rates compared to conventional methods.
Survey on uncertainty in GNNs for improved model reliability.
problem Inherent randomness and model training errors lead to unstable GNN predictions.
method Comprehensive overview of GNN uncertainty theory and methods.
result Integration of uncertainty in GNNs enhances model performance and reliability.
Paper establishes a comprehensive benchmark for ECG time-series analysis.
problem Incomplete understanding of ECG signal properties and limitations in evaluation metrics.
method Categorization of downstream applications, identification of limitations, introduction of a novel metric, benchmarking of time-series models.
result Validation of the effectiveness of the proposed metric and model architecture.
New method controls bias in training data for fair outcomes.
problem Ensuring equal treatment between different groups in machine learning.
method Contrastive information estimation to control mutual information between representations and protected attributes.
result Our method provides strong theoretical guarantees on the parity of any downstream algorithm.
New method predicts wind farm power and wakes using weather patterns.
problem Inefficient and computationally intensive wind energy resource assessment.
method Unsupervised clustering of ERA5 data on wind velocity, WRF simulations at cluster centers, and post-processing.
result Accurate long-term predictions of power and wakes with reduced computational time.
New methods reduce bias in synthetic data for machine learning.
problem Statistical bias in synthetic data generated for privacy.
method Re-weighting strategies using privatised likelihood ratios.
result Private importance weighting enhances synthetic data utility.
Theoretical analysis shows pretext-based self-supervised learning can be boosted by downstream data under certain conditions.
problem Theoretical analysis of pretext-based self-supervised learning and downstream data refinement.
method Theoretical analysis and experiments on synthetic and real-world datasets.
result Theoretical lower bounds and experiments show that downstream data refinement can boost or hurt performance depending on conditions.
This study examines how the size and alignment of pretraining data affect the performance of large language models on downstream tasks.
problem Understanding how the size and alignment of pretraining data impact the performance of large language models on downstream tasks.
method Investigated the scaling behavior of large language models in a transfer learning setting, focusing on machine translation tasks.
result The size of the finetuning dataset and the distribution alignment between pretraining and downstream data significantly influence the scaling behavior of downstream performance.
COBRA reduces modality gap in cross-modal tasks.
problem Joint embedding spaces fail to sufficiently reduce modality gap in multi-modal tasks.
method COBRA trains image and text modalities in a joint fashion using Contrastive Predictive Coding and Noise Contrastive Estimation.
result COBRA significantly reduces the modality gap and generates robust joint-embedding space.
Self-supervised learning improves few-shot classification and segmentation on point clouds.
problem Efficiently learn from limited labeled data in point cloud applications.
method Hierarchical cover-tree partitioning for self-supervised pre-training; restricted to support set for few-shot learning.
result Self-supervised learning significantly improves downstream classification and segmentation accuracy.
New algorithm balances spatial data approximation and prediction accuracy.
problem Lack of methods considering spatial correlation and downstream modeling in dimension reduction.
method Formalizes approximation and modeling utility as metrics, proposes a balanced algorithm.
result Optimal trade-off between approximation accuracy and downstream modeling utility.
Paper presents AETN for efficient user modeling from mobile app usage.
problem Efficient user modeling from mobile app usage with reduced manual effort.
method AutoEncoder-coupled Transformer Network (AETN).
result AETN achieves effective user embeddings with reduced manual effort.
Paper defines and solves a problem in representation learning to ensure fairness with high confidence.
problem Learning fair representations with high confidence guarantees for all downstream tasks.
method Formally defines the problem, introduces FRG framework, proves high probability fairness, and demonstrates effectiveness empirically.
result FRG framework provides high-confidence guarantees for limiting unfairness across all downstream models and tasks.
PEARL combines multiple representation learning methods to enhance model performance.
problem Different representation learning methods extract distinct data aspects, potentially missing important insights.
method Combines multiple representation learning approaches using surrogate loss functions for efficient weight estimation.
result Asymptotically achieves optimal performance in downstream tasks, assigning nonzero weights to correctly specified models.
As machine learning (ML) increasingly affects people and society, awareness of its potential unwanted consequences has also grown. To anticipate, prevent, and mitigate undesirable downstream consequences, it is critical that we understand when and how harm might be introduced throughout the ML life cycle. In this paper…
Self-supervised metric learning boosts downstream tasks in multi-view data.
problem Improving distance-based downstream tasks without labeled data.
method Developed a statistical framework to study self-supervised metric learning in multi-view data.
result Self-supervised metric learning improves target distances for various downstream tasks.
Study shows scaling up models doesn't always improve downstream tasks.
problem Understanding why scaling up models doesn't always improve downstream performance.
method Systematic study of 4800 experiments on various models, analyzing performance on 20 downstream tasks.
result Performance on downstream tasks saturates as model size increases, revealing a nonlinear relationship.
PRAGMA models financial event sequences for various banking tasks.
problem Handling diverse financial data for multiple applications.
method Pre-training a Transformer model on a large banking event corpus with a self-supervised objective.
result PRAGMA achieves superior performance across multiple financial domains from raw event sequences.
Unsupervised learning models can produce accurate but misleading predictions.
problem Widespread misleading predictions in unsupervised learning models.
method Developed Explainable AI techniques to detect misleading predictions.
result Widespread Clever Hans effects in unsupervised learning models.
Analysis of pretrained models' effectiveness in downstream tasks.
problem Understanding why pretrained models perform well in NLP tasks.
method Analyzed head and prompt tuning approaches using latent variable models.
result Prompt tuning provides stronger guarantees than head tuning.
New method uses imperfect LLM annotations for valid statistical inference in social science.
problem Inaccurate large language model annotations in social science research.
method Design-based supervised learning (DSL) combining imperfect LLM surrogates with gold-standard labels.
result DSL provides valid statistical inference with comparable predictive accuracy to existing methods.
Study shows disentanglement models learn correlations from data, impacting fairness.
problem Disentanglement models learn correlations in real-world data, affecting downstream applications.
method Empirical study on 4260 models, analyzing correlations in latent representations.
result Systematically induced correlations are learned by disentanglement models, impacting fairness.
Introduces lookahead counterfactual fairness to account for downstream effects of ML predictions.
problem Downstream effects of ML predictions on individuals not considered by counterfactual fairness.
method Introduces lookahead counterfactual fairness (LCF), a new fairness notion that considers future status. Proposes an algorithm based on theoretical conditions.
result Proposes an algorithm to achieve lookahead counterfactual fairness and validates it on synthetic and real data.
Geospatial ML models need special evaluation methods due to their unique challenges.
problem Evaluating geospatial machine learning models is challenging due to their specific characteristics.
method Delineated unique challenges and proposed concrete takeaways for improving geospatial model evaluations.
result Concrete takeaways for improving evaluations of geospatial model performance.
A central goal of unsupervised learning is to acquire representations from unlabeled data or experience that can be used for more effective learning of downstream tasks from modest amounts of labeled data. Many prior unsupervised learning works aim to do so by developing proxy objectives based on reconstruction, disent…
Graph neural networks (GNNs) are shown to be successful in modeling applications with graph structures. However, training an accurate GNN model requires a large collection of labeled data and expressive features, which might be inaccessible for some applications. To tackle this problem, we propose a pre-training framew…
This paper characterizes VAE training pathologies and their effects on tasks.
problem Characterizing VAE training pathologies and their impact on downstream tasks.
method Concretely characterizing conditions for VAE training pathologies and their connection to specific downstream tasks.
result Connects VAE training pathologies to specific downstream tasks like learning compressed and disentangled representations, adversarial robustness, and semi-supervised learning.
This work uses conformal prediction to quantify uncertainty in large language models for multiple-choice questions.
problem Ensuring robustness and reliability of large language models in high-stakes applications.
method Conformal prediction applied to multi-choice question answering.
result Uncertainty estimates from conformal prediction are closely related to prediction accuracy.
Develops a statistical framework for self-supervised representation learning using data augmentation.
problem Lack of theoretical understanding of data augmentation in nonlinear settings.
method Augmentation invariant manifold learning framework and stochastic optimization algorithm.
result Improves downstream analysis by exploiting manifold's geometric structure and invariant property of augmented data.
New framework explains how larger pre-trained models reduce downstream learning sample complexity.
problem Understanding why larger pre-trained models reduce sample complexity in downstream tasks.
method Introducing a novel framework called Caulking inspired by PEFT methods.
result Improved pre-trained models provably decrease downstream task sample complexity.
New method corrects bias in density ratio estimation for missing data.
problem Missing data bias in density ratio estimation.
method Adapted KLIEP method (M-KLIEP) for MNAR data.
result M-KLIEP restores consistency and minimax optimality.
Bayesian graph contrastive learning improves uncertainty quantification for graph analytics.
problem Uncertainty quantification for node representations in graph contrastive learning.
method Proposes a Bayesian framework to learn stochastic encoders representing nodes as distributions, providing uncertainty estimates.
result Significant improvement in performance on benchmark datasets compared to existing methods.
Bayesian neural networks (BNNs) allow us to reason about uncertainty in a principled way. Stochastic Gradient Langevin Dynamics (SGLD) enables efficient BNN learning by drawing samples from the BNN posterior using mini-batches. However, SGLD and its extensions require storage of many copies of the model parameters, a p…
Diffusion models generate private synthetic images with high quality.
problem Generating private synthetic images from sensitive datasets.
method Differentially private fine-tuning of pre-trained diffusion models.
result Improved synthetic data quality and accuracy compared to non-private models.
A new method for experimental design focuses on predicting downstream quantities of interest.
problem Designs that maximize parameter learning may not maximize downstream quantity prediction.
method Likelihood-free goal-oriented optimal experimental design (LF-GO-OED) using ABC density ratio estimation.
result LF-GO-OED maximizes the expected information gain for downstream quantities.
UBM transfers bias mitigation from upstream to downstream tasks efficiently.
problem Bias in fine-tuned language models across various tasks.
method Apply bias mitigation to an upstream model, then fine-tune a downstream model on this mitigated model.
result UBM effects transfer to new downstream tasks, creating less biased models.
Mask-reconstruction pretraining helps in downstream tasks by capturing more semantic features.
problem How mask-reconstruction pretraining helps in downstream tasks and why it surpasses supervised learning.
method Theoretical analysis and experimental validation of mask-reconstruction pretraining (MRP) on auto-encoders.
result MRP provably captures more semantic features than supervised learning, leading to better performance in downstream tasks.
Enhances performance on downstream tasks using multi-domain data.
problem Improving performance on tasks with limited downstream data.
method Deep transfer learning framework leveraging shared and domain-specific features.
result Significantly improves convergence rate for learning Lipschitz functions.
MBExplainer provides explanations for models combining graph embeddings and tabular features.
problem Explaining models using a mix of graph embeddings and tabular features.
method Model-agnostic approach using Shapley values and Monte Carlo Tree Search.
result MBExplainer efficiently finds human-readable explanations for model predictions.
Improved RL for knowledge graph reasoning with entity types.
problem Challenges in path-based relational reasoning over knowledge graphs.
method Type-enhanced RL agent using GNN for neighborhood information.
result Outperforms state-of-the-art RL methods and discovers novel paths.
New method improves subspace iteration for eigenvectors in machine learning.
problem Computing eigenvectors for large-scale problems in machine learning.
method Subspace iteration with ℓ2o∞ norm convergence analysis. result Deterministic bounds and practical stopping criterion for improved performance.
ACERL embeds networks into a low-dimensional space preserving structural and semantic properties.
problem Challenges in brain connectivity data analysis with subject-specific, high-dimensional, and sparse networks.
method Contrastive learning of augmented network pairs with adaptive random masking.
result Achieves minimax optimal convergence rate for edge representation learning.
This work optimizes alignment and uniformity of features on a hypersphere for better downstream performance.
problem Improving the performance of contrastive representation learning.
method Identifying and optimizing alignment and uniformity of features on a hypersphere.
result Directly optimizing alignment and uniformity leads to comparable or better performance than contrastive learning.
Proposes a new model for joint probability distributions in computer vision.
problem Limitation of existing models in meeting diverse downstream tasks.
method Uses parametric conditional probability distributions for each group of variables conditioned on the rest.
result Models can be used for any downstream task without task-specific design.